2023-07-14T17:17:12.812 INFO:root:teuthology version: 0.0.1.dev145+gaa6a80e 2023-07-14T17:17:12.813 DEBUG:teuthology.run:Teuthology command: teuthology --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}} --archive /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337974 --owner scheduled_pdonnell@teuthology --name pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi --verbose -- /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337974/orig.config.yaml 2023-07-14T17:17:12.867 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-14T17:17:12.956 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337974 branch: wip-pdonnell-testing-20230714.134335 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}} email: pdonnell@redhat.com first_in_suite: false job_id: '7337974' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20230714.134335 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt dry_run: true install: ceph: flavor: default sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec kclient: client.0: mount_subvol_num: 1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20230714.134335 sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5179 sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec sleep_before_teardown: 0 subset: 285/512 suite: fs:workload suite_branch: wip-pdonnell-testing-20230714.134335 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi088.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi107.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi178.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs _nogroup random 0.01 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - workunit: clients: all: - direct_io teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: aa6a80e6cc655fff48452a7eca02070e61d074fc timestamp: 2023-07-14_16:44:47 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-14T17:17:12.956 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa; will attempt to use it 2023-07-14T17:17:12.957 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa/tasks 2023-07-14T17:17:12.957 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-14T17:17:12.959 INFO:teuthology.task.internal:Checking packages... 2023-07-14T17:17:12.988 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '81c9387d2e7576da065747ec9c16d1c7d69016ec' 2023-07-14T17:17:12.988 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-14T17:17:12.989 INFO:teuthology.packaging:ref: None 2023-07-14T17:17:12.989 INFO:teuthology.packaging:tag: None 2023-07-14T17:17:12.989 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20230714.134335 2023-07-14T17:17:12.989 INFO:teuthology.packaging:sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T17:17:12.990 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20230714.134335 2023-07-14T17:17:13.521 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-4940.g81c9387d 2023-07-14T17:17:13.521 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-14T17:17:13.528 INFO:teuthology.task.internal:no buildpackages task found 2023-07-14T17:17:13.528 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-14T17:17:13.534 INFO:teuthology.task.internal:Saving configuration 2023-07-14T17:17:13.567 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-14T17:17:13.573 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T17:17:18.607 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337974', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:08:25.060848', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T17:17:28.643 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337974', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:08:25.061798', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T17:17:33.675 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337974', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:08:25.059060', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T17:17:33.676 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-14T17:17:33.682 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-07-14T17:17:33.682 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-07-14T17:17:33.683 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-07-14T17:17:33.683 INFO:teuthology.run_tasks:Running task console_log... 2023-07-14T17:17:48.811 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f3d390fd3a0>, signals=[15]) 2023-07-14T17:17:48.813 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-14T17:17:48.825 INFO:teuthology.task.internal:Opening connections... 2023-07-14T17:17:48.826 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2023-07-14T17:17:48.829 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T17:17:48.899 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2023-07-14T17:17:48.900 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T17:17:53.972 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi178.front.sepia.ceph.com 2023-07-14T17:17:53.973 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T17:17:54.042 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-14T17:17:54.053 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2023-07-14T17:17:54.074 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2023-07-14T17:17:54.075 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2023-07-14T17:17:54.134 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T17:17:54.134 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T17:17:54.134 INFO:teuthology.orchestra.run.smithi088.stdout:ID="rhel" 2023-07-14T17:17:54.134 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="fedora" 2023-07-14T17:17:54.135 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="8.6" 2023-07-14T17:17:54.135 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el8" 2023-07-14T17:17:54.135 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T17:17:54.135 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2023-07-14T17:17:54.135 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T17:17:54.135 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T17:17:54.135 INFO:teuthology.orchestra.run.smithi088.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T17:17:54.136 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T17:17:54.136 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:17:54.136 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T17:17:54.136 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T17:17:54.136 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T17:17:54.136 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T17:17:54.137 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2023-07-14T17:17:54.157 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2023-07-14T17:17:54.177 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2023-07-14T17:17:54.177 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2023-07-14T17:17:54.235 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T17:17:54.235 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T17:17:54.235 INFO:teuthology.orchestra.run.smithi107.stdout:ID="rhel" 2023-07-14T17:17:54.236 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="fedora" 2023-07-14T17:17:54.236 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="8.6" 2023-07-14T17:17:54.236 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el8" 2023-07-14T17:17:54.236 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T17:17:54.236 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2023-07-14T17:17:54.236 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T17:17:54.237 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T17:17:54.237 INFO:teuthology.orchestra.run.smithi107.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T17:17:54.237 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T17:17:54.237 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-07-14T17:17:54.237 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T17:17:54.237 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T17:17:54.237 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T17:17:54.238 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T17:17:54.238 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2023-07-14T17:17:54.257 DEBUG:teuthology.orchestra.run.smithi178:> uname -m 2023-07-14T17:17:54.277 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64 2023-07-14T17:17:54.278 DEBUG:teuthology.orchestra.run.smithi178:> cat /etc/os-release 2023-07-14T17:17:54.335 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T17:17:54.336 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T17:17:54.336 INFO:teuthology.orchestra.run.smithi178.stdout:ID="rhel" 2023-07-14T17:17:54.336 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE="fedora" 2023-07-14T17:17:54.336 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="8.6" 2023-07-14T17:17:54.336 INFO:teuthology.orchestra.run.smithi178.stdout:PLATFORM_ID="platform:el8" 2023-07-14T17:17:54.336 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T17:17:54.337 INFO:teuthology.orchestra.run.smithi178.stdout:ANSI_COLOR="0;31" 2023-07-14T17:17:54.337 INFO:teuthology.orchestra.run.smithi178.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T17:17:54.337 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T17:17:54.337 INFO:teuthology.orchestra.run.smithi178.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T17:17:54.337 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T17:17:54.337 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-07-14T17:17:54.337 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T17:17:54.337 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T17:17:54.338 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T17:17:54.338 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T17:17:54.338 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server 2023-07-14T17:17:54.374 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-14T17:17:54.388 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-14T17:17:54.394 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-14T17:17:54.395 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2023-07-14T17:17:54.398 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2023-07-14T17:17:54.401 DEBUG:teuthology.orchestra.run.smithi178:> test '!' -e /home/ubuntu/cephtest 2023-07-14T17:17:54.418 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-14T17:17:54.425 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-14T17:17:54.425 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2023-07-14T17:17:54.458 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2023-07-14T17:17:54.461 DEBUG:teuthology.orchestra.run.smithi178:> test -z $(ls -A /var/lib/ceph) 2023-07-14T17:17:54.488 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-14T17:17:54.576 INFO:teuthology.run_tasks:Running task kernel... 2023-07-14T17:17:54.592 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-07-14T17:17:54.593 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-07-14T17:17:54.593 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T17:17:54.594 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T17:17:54.595 DEBUG:teuthology.orchestra.run.smithi178:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T17:17:54.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:17:54.612 DEBUG:teuthology.orchestra.run.smithi178:> uname -r 2023-07-14T17:17:54.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:17:54.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:17:54.614 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2023-07-14T17:17:54.614 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2023-07-14T17:17:54.669 INFO:teuthology.orchestra.run.smithi178.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-14T17:17:54.671 INFO:teuthology.task.kernel:Running kernel on smithi178: 4.18.0-372.9.1.el8.x86_64 2023-07-14T17:17:54.671 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum install -y kernel 2023-07-14T17:17:54.672 INFO:teuthology.orchestra.run.smithi088.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-14T17:17:54.673 INFO:teuthology.task.kernel:Running kernel on smithi088: 4.18.0-372.9.1.el8.x86_64 2023-07-14T17:17:54.673 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum install -y kernel 2023-07-14T17:17:54.674 INFO:teuthology.orchestra.run.smithi107.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-14T17:17:54.674 INFO:teuthology.task.kernel:Running kernel on smithi107: 4.18.0-372.9.1.el8.x86_64 2023-07-14T17:17:54.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2023-07-14T17:17:55.813 INFO:teuthology.orchestra.run.smithi178.stdout:Updating Subscription Management repositories. 2023-07-14T17:17:55.814 INFO:teuthology.orchestra.run.smithi178.stdout:Unable to read consumer identity 2023-07-14T17:17:55.881 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-07-14T17:17:55.881 INFO:teuthology.orchestra.run.smithi088.stdout:Unable to read consumer identity 2023-07-14T17:17:55.884 INFO:teuthology.orchestra.run.smithi107.stdout:Updating Subscription Management repositories. 2023-07-14T17:17:55.885 INFO:teuthology.orchestra.run.smithi107.stdout:Unable to read consumer identity 2023-07-14T17:17:57.439 INFO:teuthology.orchestra.run.smithi088.stdout:Copr repo for python3-asyncssh owned by ceph 10 kB/s | 3.5 kB 00:00 2023-07-14T17:17:57.491 INFO:teuthology.orchestra.run.smithi178.stdout:Copr repo for python3-asyncssh owned by ceph 11 kB/s | 3.5 kB 00:00 2023-07-14T17:17:57.622 INFO:teuthology.orchestra.run.smithi107.stdout:Copr repo for python3-asyncssh owned by ceph 11 kB/s | 3.5 kB 00:00 2023-07-14T17:18:09.470 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 1.3 MB/s | 16 MB 00:11 2023-07-14T17:18:12.945 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 318 kB/s | 24 kB 00:00 2023-07-14T17:18:13.656 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-14T17:18:13.719 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-07-14T17:18:13.720 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2023-07-14T17:18:13.720 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-07-14T17:18:13.844 DEBUG:teuthology.orchestra.run.smithi107:> echo no | sudo yum reinstall kernel || true 2023-07-14T17:18:14.177 INFO:teuthology.orchestra.run.smithi107.stdout:Updating Subscription Management repositories. 2023-07-14T17:18:14.177 INFO:teuthology.orchestra.run.smithi107.stdout:Unable to read consumer identity 2023-07-14T17:18:14.441 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:02 ago on Fri 14 Jul 2023 05:18:12 PM UTC. 2023-07-14T17:18:14.482 INFO:teuthology.orchestra.run.smithi107.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T17:18:14.510 INFO:teuthology.orchestra.run.smithi107.stderr:Error: No packages marked for reinstall. 2023-07-14T17:18:14.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum reinstall -y kernel || true 2023-07-14T17:18:14.882 INFO:teuthology.orchestra.run.smithi107.stdout:Updating Subscription Management repositories. 2023-07-14T17:18:14.882 INFO:teuthology.orchestra.run.smithi107.stdout:Unable to read consumer identity 2023-07-14T17:18:15.153 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:03 ago on Fri 14 Jul 2023 05:18:12 PM UTC. 2023-07-14T17:18:15.193 INFO:teuthology.orchestra.run.smithi107.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T17:18:15.215 INFO:teuthology.orchestra.run.smithi107.stderr:Error: No packages marked for reinstall. 2023-07-14T17:18:15.261 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2023-07-14T17:18:15.341 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-14T17:18:15.342 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-14T17:18:15.342 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-14T17:18:15.342 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-07-14T17:18:15.342 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-14T17:18:15.376 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2023-07-14T17:18:15.389 DEBUG:teuthology.parallel:result is None 2023-07-14T17:18:26.487 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 551 kB/s | 16 MB 00:28 2023-07-14T17:18:29.331 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 501 kB/s | 16 MB 00:31 2023-07-14T17:18:31.452 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 15 kB/s | 24 kB 00:01 2023-07-14T17:18:32.146 INFO:teuthology.orchestra.run.smithi178.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-14T17:18:32.205 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-07-14T17:18:32.206 INFO:teuthology.orchestra.run.smithi178.stdout:Nothing to do. 2023-07-14T17:18:32.206 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-07-14T17:18:32.363 DEBUG:teuthology.orchestra.run.smithi178:> echo no | sudo yum reinstall kernel || true 2023-07-14T17:18:32.695 INFO:teuthology.orchestra.run.smithi178.stdout:Updating Subscription Management repositories. 2023-07-14T17:18:32.695 INFO:teuthology.orchestra.run.smithi178.stdout:Unable to read consumer identity 2023-07-14T17:18:32.730 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 454 kB/s | 24 kB 00:00 2023-07-14T17:18:32.967 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:01 ago on Fri 14 Jul 2023 05:18:31 PM UTC. 2023-07-14T17:18:33.007 INFO:teuthology.orchestra.run.smithi178.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T17:18:33.048 INFO:teuthology.orchestra.run.smithi178.stderr:Error: No packages marked for reinstall. 2023-07-14T17:18:33.095 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum reinstall -y kernel || true 2023-07-14T17:18:33.416 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-14T17:18:33.423 INFO:teuthology.orchestra.run.smithi178.stdout:Updating Subscription Management repositories. 2023-07-14T17:18:33.424 INFO:teuthology.orchestra.run.smithi178.stdout:Unable to read consumer identity 2023-07-14T17:18:33.474 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-07-14T17:18:33.475 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2023-07-14T17:18:33.475 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-07-14T17:18:33.598 DEBUG:teuthology.orchestra.run.smithi088:> echo no | sudo yum reinstall kernel || true 2023-07-14T17:18:33.685 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:00:02 ago on Fri 14 Jul 2023 05:18:31 PM UTC. 2023-07-14T17:18:33.724 INFO:teuthology.orchestra.run.smithi178.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T17:18:33.771 INFO:teuthology.orchestra.run.smithi178.stderr:Error: No packages marked for reinstall. 2023-07-14T17:18:33.817 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q kernel | sort -rV | head -n 1 2023-07-14T17:18:33.897 INFO:teuthology.orchestra.run.smithi178.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-14T17:18:33.897 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-14T17:18:33.897 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-14T17:18:33.897 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2023-07-14T17:18:33.898 DEBUG:teuthology.orchestra.run.smithi178:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-14T17:18:33.924 INFO:teuthology.orchestra.run.smithi178.stdout:ttyS1 2023-07-14T17:18:33.927 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-07-14T17:18:33.927 INFO:teuthology.orchestra.run.smithi088.stdout:Unable to read consumer identity 2023-07-14T17:18:33.945 DEBUG:teuthology.parallel:result is None 2023-07-14T17:18:34.209 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:02 ago on Fri 14 Jul 2023 05:18:32 PM UTC. 2023-07-14T17:18:34.243 INFO:teuthology.orchestra.run.smithi088.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T17:18:34.282 INFO:teuthology.orchestra.run.smithi088.stderr:Error: No packages marked for reinstall. 2023-07-14T17:18:34.327 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum reinstall -y kernel || true 2023-07-14T17:18:34.657 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-07-14T17:18:34.657 INFO:teuthology.orchestra.run.smithi088.stdout:Unable to read consumer identity 2023-07-14T17:18:34.916 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:02 ago on Fri 14 Jul 2023 05:18:32 PM UTC. 2023-07-14T17:18:34.955 INFO:teuthology.orchestra.run.smithi088.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T17:18:34.989 INFO:teuthology.orchestra.run.smithi088.stderr:Error: No packages marked for reinstall. 2023-07-14T17:18:35.037 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2023-07-14T17:18:35.116 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-14T17:18:35.117 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-14T17:18:35.117 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-14T17:18:35.117 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-07-14T17:18:35.117 DEBUG:teuthology.orchestra.run.smithi088:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-14T17:18:35.149 INFO:teuthology.orchestra.run.smithi088.stdout:ttyS1 2023-07-14T17:18:35.162 DEBUG:teuthology.parallel:result is None 2023-07-14T17:18:35.163 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-14T17:18:35.169 INFO:teuthology.task.internal:Creating test directory... 2023-07-14T17:18:35.169 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T17:18:35.204 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T17:18:35.207 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T17:18:35.227 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-14T17:18:35.238 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-14T17:18:35.245 INFO:teuthology.task.internal:Creating archive directory... 2023-07-14T17:18:35.245 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T17:18:35.260 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T17:18:35.267 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T17:18:35.304 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-14T17:18:35.310 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-14T17:18:35.311 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T17:18:35.317 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T17:18:35.339 DEBUG:teuthology.orchestra.run.smithi178:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T17:18:35.367 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T17:18:35.378 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T17:18:35.403 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T17:18:35.414 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T17:18:35.414 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T17:18:35.426 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T17:18:35.428 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-14T17:18:35.436 INFO:teuthology.task.internal:Configuring sudo... 2023-07-14T17:18:35.437 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T17:18:35.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T17:18:35.441 DEBUG:teuthology.orchestra.run.smithi178:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T17:18:35.472 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-14T17:18:35.481 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-14T17:18:35.481 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T17:18:35.506 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T17:18:35.510 DEBUG:teuthology.orchestra.run.smithi178:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T17:18:35.530 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T17:18:37.772 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T17:18:37.815 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T17:18:37.904 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T17:18:37.947 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:18:37.948 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T17:18:38.017 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T17:18:38.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T17:18:38.122 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T17:18:38.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T17:18:38.252 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-07-14T17:18:38.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T17:18:38.321 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T17:18:38.372 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T17:18:38.426 DEBUG:teuthology.orchestra.run.smithi178:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T17:18:38.517 DEBUG:teuthology.orchestra.run.smithi178:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T17:18:38.558 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-07-14T17:18:38.558 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T17:18:38.627 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2023-07-14T17:18:38.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2023-07-14T17:18:38.635 DEBUG:teuthology.orchestra.run.smithi178:> sudo service rsyslog restart 2023-07-14T17:18:38.690 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T17:18:38.704 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T17:18:38.723 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T17:18:39.019 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-14T17:18:39.027 INFO:teuthology.task.internal:Starting timer... 2023-07-14T17:18:39.028 INFO:teuthology.run_tasks:Running task pcp... 2023-07-14T17:18:39.038 INFO:teuthology.run_tasks:Running task selinux... 2023-07-14T17:18:39.052 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-07-14T17:18:39.053 DEBUG:teuthology.orchestra.run.smithi088:> sudo service auditd rotate 2023-07-14T17:18:39.115 INFO:teuthology.orchestra.run.smithi088.stdout:Rotating logs: 2023-07-14T17:18:39.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2023-07-14T17:18:39.186 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2023-07-14T17:18:39.188 DEBUG:teuthology.orchestra.run.smithi178:> sudo service auditd rotate 2023-07-14T17:18:39.253 INFO:teuthology.orchestra.run.smithi178.stdout:Rotating logs: 2023-07-14T17:18:39.255 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-14T17:18:39.255 DEBUG:teuthology.orchestra.run.smithi088:> /usr/sbin/getenforce 2023-07-14T17:18:39.275 INFO:teuthology.orchestra.run.smithi088.stdout:Permissive 2023-07-14T17:18:39.275 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2023-07-14T17:18:39.311 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2023-07-14T17:18:39.311 DEBUG:teuthology.orchestra.run.smithi178:> /usr/sbin/getenforce 2023-07-14T17:18:39.329 INFO:teuthology.orchestra.run.smithi178.stdout:Permissive 2023-07-14T17:18:39.330 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi088.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi107.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi178.front.sepia.ceph.com': 'permissive'} 2023-07-14T17:18:39.330 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T17:18:39.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:18:39.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T17:18:39.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:18:39.384 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T17:18:39.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:18:39.413 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-14T17:18:39.413 DEBUG:teuthology.orchestra.run.smithi088:> sudo /usr/sbin/setenforce permissive 2023-07-14T17:18:39.440 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2023-07-14T17:18:39.467 DEBUG:teuthology.orchestra.run.smithi178:> sudo /usr/sbin/setenforce permissive 2023-07-14T17:18:39.495 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-14T17:18:39.506 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-07-14T17:18:39.514 INFO:teuthology.repo_utils:Fetching main from origin 2023-07-14T17:18:44.656 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-14T17:18:44.680 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-14T17:18:44.682 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi088.front.sepia.ceph.com,smithi107.front.sepia.ceph.com,smithi178.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-14T17:30:53.310 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi088.front.sepia.ceph.com'), Remote(name='ubuntu@smithi107.front.sepia.ceph.com'), Remote(name='ubuntu@smithi178.front.sepia.ceph.com')] 2023-07-14T17:30:53.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-07-14T17:30:53.336 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T17:30:58.408 DEBUG:teuthology.orchestra.run.smithi088:> true 2023-07-14T17:30:58.479 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-07-14T17:30:58.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-07-14T17:30:58.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T17:31:03.552 DEBUG:teuthology.orchestra.run.smithi107:> true 2023-07-14T17:31:03.623 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2023-07-14T17:31:03.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com' 2023-07-14T17:31:03.624 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi178.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T17:31:03.692 DEBUG:teuthology.orchestra.run.smithi178:> true 2023-07-14T17:31:03.762 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi178.front.sepia.ceph.com' 2023-07-14T17:31:03.763 INFO:teuthology.run_tasks:Running task clock... 2023-07-14T17:31:03.774 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-14T17:31:03.775 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T17:31:03.775 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T17:31:03.778 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T17:31:03.778 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T17:31:03.781 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T17:31:03.781 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T17:31:03.812 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T17:31:03.814 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T17:31:03.829 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T17:31:03.830 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T17:31:03.838 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T17:31:03.853 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T17:31:03.897 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: ntpd: command not found 2023-07-14T17:31:03.906 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2023-07-14T17:31:03.909 INFO:teuthology.orchestra.run.smithi088.stdout:506 Cannot talk to daemon 2023-07-14T17:31:03.919 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2023-07-14T17:31:03.923 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T17:31:03.928 INFO:teuthology.orchestra.run.smithi178.stderr:sudo: ntpd: command not found 2023-07-14T17:31:03.933 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T17:31:03.938 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T17:31:03.940 INFO:teuthology.orchestra.run.smithi178.stdout:506 Cannot talk to daemon 2023-07-14T17:31:03.948 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T17:31:03.956 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T17:31:03.970 INFO:teuthology.orchestra.run.smithi178.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T17:31:04.001 INFO:teuthology.orchestra.run.smithi088.stderr:bash: ntpq: command not found 2023-07-14T17:31:04.005 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T17:31:04.005 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2023-07-14T17:31:04.006 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:31:04.006 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:31:04.006 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:31:04.006 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:31:04.021 INFO:teuthology.orchestra.run.smithi107.stderr:bash: ntpq: command not found 2023-07-14T17:31:04.025 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T17:31:04.025 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2023-07-14T17:31:04.025 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:31:04.025 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:31:04.025 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:31:04.025 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:31:04.045 INFO:teuthology.orchestra.run.smithi178.stderr:bash: ntpq: command not found 2023-07-14T17:31:04.049 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T17:31:04.049 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2023-07-14T17:31:04.049 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:31:04.049 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:31:04.049 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:31:04.050 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:31:04.051 INFO:teuthology.run_tasks:Running task pexec... 2023-07-14T17:31:04.059 INFO:teuthology.task.pexec:Executing custom commands... 2023-07-14T17:31:04.060 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi088.front.sepia.ceph.com 2023-07-14T17:31:04.061 DEBUG:teuthology.orchestra.run.smithi088:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T17:31:04.061 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi107.front.sepia.ceph.com 2023-07-14T17:31:04.061 DEBUG:teuthology.orchestra.run.smithi107:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T17:31:04.062 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi178.front.sepia.ceph.com 2023-07-14T17:31:04.062 DEBUG:teuthology.orchestra.run.smithi178:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T17:31:04.431 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-07-14T17:31:04.435 INFO:teuthology.orchestra.run.smithi107.stdout:Updating Subscription Management repositories. 2023-07-14T17:31:04.465 INFO:teuthology.orchestra.run.smithi178.stdout:Updating Subscription Management repositories. 2023-07-14T17:31:05.573 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:03:31 ago on Fri 14 Jul 2023 05:27:34 PM UTC. 2023-07-14T17:31:05.575 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:03:51 ago on Fri 14 Jul 2023 05:27:14 PM UTC. 2023-07-14T17:31:05.874 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:38 ago on Fri 14 Jul 2023 05:29:27 PM UTC. 2023-07-14T17:31:07.326 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-07-14T17:31:07.327 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-07-14T17:31:07.327 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2023-07-14T17:31:07.327 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-07-14T17:31:07.327 INFO:teuthology.orchestra.run.smithi107.stdout:Resetting modules: 2023-07-14T17:31:07.327 INFO:teuthology.orchestra.run.smithi107.stdout: container-tools 2023-07-14T17:31:07.327 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-07-14T17:31:07.328 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-07-14T17:31:07.328 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2023-07-14T17:31:07.328 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-07-14T17:31:07.350 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-07-14T17:31:07.351 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-07-14T17:31:07.351 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2023-07-14T17:31:07.351 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-07-14T17:31:07.352 INFO:teuthology.orchestra.run.smithi088.stdout:Resetting modules: 2023-07-14T17:31:07.352 INFO:teuthology.orchestra.run.smithi088.stdout: container-tools 2023-07-14T17:31:07.352 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:31:07.352 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-07-14T17:31:07.352 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-07-14T17:31:07.352 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:31:07.603 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-07-14T17:31:07.604 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-07-14T17:31:07.604 INFO:teuthology.orchestra.run.smithi178.stdout: Package Architecture Version Repository Size 2023-07-14T17:31:07.604 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-07-14T17:31:07.605 INFO:teuthology.orchestra.run.smithi178.stdout:Resetting modules: 2023-07-14T17:31:07.605 INFO:teuthology.orchestra.run.smithi178.stdout: container-tools 2023-07-14T17:31:07.605 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-07-14T17:31:07.605 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-07-14T17:31:07.605 INFO:teuthology.orchestra.run.smithi178.stdout:================================================================================ 2023-07-14T17:31:07.605 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-07-14T17:31:08.283 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:03:33 ago on Fri 14 Jul 2023 05:27:34 PM UTC. 2023-07-14T17:31:08.327 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:03:53 ago on Fri 14 Jul 2023 05:27:14 PM UTC. 2023-07-14T17:31:08.568 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:41 ago on Fri 14 Jul 2023 05:29:27 PM UTC. 2023-07-14T17:31:14.714 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-07-14T17:31:15.160 INFO:teuthology.orchestra.run.smithi178.stdout:Updating Subscription Management repositories. 2023-07-14T17:31:16.814 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:49 ago on Fri 14 Jul 2023 05:29:27 PM UTC. 2023-07-14T17:31:17.223 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-07-14T17:31:17.679 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-07-14T17:31:18.047 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-07-14T17:31:18.491 INFO:teuthology.orchestra.run.smithi107.stdout:Updating Subscription Management repositories. 2023-07-14T17:31:18.514 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-07-14T17:31:18.603 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================================ 2023-07-14T17:31:18.603 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-07-14T17:31:18.603 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================================ 2023-07-14T17:31:18.603 INFO:teuthology.orchestra.run.smithi178.stdout:Installing group/module packages: 2023-07-14T17:31:18.604 INFO:teuthology.orchestra.run.smithi178.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T17:31:18.604 INFO:teuthology.orchestra.run.smithi178.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T17:31:18.604 INFO:teuthology.orchestra.run.smithi178.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T17:31:18.604 INFO:teuthology.orchestra.run.smithi178.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T17:31:18.604 INFO:teuthology.orchestra.run.smithi178.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T17:31:18.604 INFO:teuthology.orchestra.run.smithi178.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T17:31:18.605 INFO:teuthology.orchestra.run.smithi178.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T17:31:18.605 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-07-14T17:31:18.605 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T17:31:18.605 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T17:31:18.605 INFO:teuthology.orchestra.run.smithi178.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T17:31:18.606 INFO:teuthology.orchestra.run.smithi178.stdout:Installing module profiles: 2023-07-14T17:31:18.606 INFO:teuthology.orchestra.run.smithi178.stdout: container-tools/common 2023-07-14T17:31:18.606 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-07-14T17:31:18.606 INFO:teuthology.orchestra.run.smithi178.stdout:Enabling module streams: 2023-07-14T17:31:18.606 INFO:teuthology.orchestra.run.smithi178.stdout: container-tools rhel8 2023-07-14T17:31:18.606 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-07-14T17:31:18.607 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-07-14T17:31:18.607 INFO:teuthology.orchestra.run.smithi178.stdout:============================================================================================================ 2023-07-14T17:31:18.607 INFO:teuthology.orchestra.run.smithi178.stdout:Install 10 Packages 2023-07-14T17:31:18.607 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-07-14T17:31:18.608 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 18 M 2023-07-14T17:31:18.608 INFO:teuthology.orchestra.run.smithi178.stdout:Installed size: 65 M 2023-07-14T17:31:18.608 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-07-14T17:31:18.782 INFO:teuthology.orchestra.run.smithi178.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 532 kB/s | 94 kB 00:00 2023-07-14T17:31:18.807 INFO:teuthology.orchestra.run.smithi178.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 202 kB/s | 41 kB 00:00 2023-07-14T17:31:18.833 INFO:teuthology.orchestra.run.smithi178.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 109 kB/s | 25 kB 00:00 2023-07-14T17:31:18.858 INFO:teuthology.orchestra.run.smithi178.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 653 kB/s | 49 kB 00:00 2023-07-14T17:31:18.958 INFO:teuthology.orchestra.run.smithi178.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.4 MB/s | 524 kB 00:00 2023-07-14T17:31:19.009 INFO:teuthology.orchestra.run.smithi178.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 994 kB/s | 149 kB 00:00 2023-07-14T17:31:19.142 INFO:teuthology.orchestra.run.smithi178.stdout:(7/10): buildah-1.26.2-1.module+el8.6.0+16771+2 26 MB/s | 8.0 MB 00:00 2023-07-14T17:31:19.196 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:04:04 ago on Fri 14 Jul 2023 05:27:14 PM UTC. 2023-07-14T17:31:19.197 INFO:teuthology.orchestra.run.smithi178.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 895 kB/s | 213 kB 00:00 2023-07-14T17:31:19.351 INFO:teuthology.orchestra.run.smithi178.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 21 MB/s | 7.0 MB 00:00 2023-07-14T17:31:19.401 INFO:teuthology.orchestra.run.smithi178.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 8.5 MB/s | 2.2 MB 00:00 2023-07-14T17:31:19.402 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-07-14T17:31:19.402 INFO:teuthology.orchestra.run.smithi178.stdout:Total 23 MB/s | 18 MB 00:00 2023-07-14T17:31:19.601 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-07-14T17:31:19.619 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-07-14T17:31:19.620 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-07-14T17:31:19.800 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-07-14T17:31:19.804 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-07-14T17:31:19.939 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:03:45 ago on Fri 14 Jul 2023 05:27:34 PM UTC. 2023-07-14T17:31:20.329 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-07-14T17:31:20.547 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T17:31:20.709 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T17:31:20.896 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-07-14T17:31:20.936 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T17:31:20.988 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================================ 2023-07-14T17:31:20.988 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2023-07-14T17:31:20.989 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================================ 2023-07-14T17:31:20.989 INFO:teuthology.orchestra.run.smithi088.stdout:Installing group/module packages: 2023-07-14T17:31:20.989 INFO:teuthology.orchestra.run.smithi088.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T17:31:20.989 INFO:teuthology.orchestra.run.smithi088.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T17:31:20.989 INFO:teuthology.orchestra.run.smithi088.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T17:31:20.990 INFO:teuthology.orchestra.run.smithi088.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T17:31:20.990 INFO:teuthology.orchestra.run.smithi088.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T17:31:20.990 INFO:teuthology.orchestra.run.smithi088.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T17:31:20.990 INFO:teuthology.orchestra.run.smithi088.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T17:31:20.990 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2023-07-14T17:31:20.990 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T17:31:20.990 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T17:31:20.991 INFO:teuthology.orchestra.run.smithi088.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T17:31:20.991 INFO:teuthology.orchestra.run.smithi088.stdout:Installing module profiles: 2023-07-14T17:31:20.991 INFO:teuthology.orchestra.run.smithi088.stdout: container-tools/common 2023-07-14T17:31:20.991 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:31:20.991 INFO:teuthology.orchestra.run.smithi088.stdout:Enabling module streams: 2023-07-14T17:31:20.991 INFO:teuthology.orchestra.run.smithi088.stdout: container-tools rhel8 2023-07-14T17:31:20.991 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:31:20.992 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-07-14T17:31:20.992 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================================ 2023-07-14T17:31:20.992 INFO:teuthology.orchestra.run.smithi088.stdout:Install 10 Packages 2023-07-14T17:31:20.992 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:31:20.993 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 18 M 2023-07-14T17:31:20.994 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 65 M 2023-07-14T17:31:20.994 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-07-14T17:31:21.169 INFO:teuthology.orchestra.run.smithi088.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 526 kB/s | 94 kB 00:00 2023-07-14T17:31:21.185 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T17:31:21.236 INFO:teuthology.orchestra.run.smithi088.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 166 kB/s | 41 kB 00:00 2023-07-14T17:31:21.261 INFO:teuthology.orchestra.run.smithi088.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 92 kB/s | 25 kB 00:00 2023-07-14T17:31:21.303 INFO:teuthology.orchestra.run.smithi088.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 369 kB/s | 49 kB 00:00 2023-07-14T17:31:21.345 INFO:teuthology.orchestra.run.smithi088.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 4.7 MB/s | 524 kB 00:00 2023-07-14T17:31:21.462 INFO:teuthology.orchestra.run.smithi088.stdout:(6/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.8 MB/s | 213 kB 00:00 2023-07-14T17:31:21.588 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T17:31:21.621 INFO:teuthology.orchestra.run.smithi088.stdout:(7/10): buildah-1.26.2-1.module+el8.6.0+16771+2 22 MB/s | 8.0 MB 00:00 2023-07-14T17:31:21.646 INFO:teuthology.orchestra.run.smithi088.stdout:(8/10): python3-podman-4.0.1-1.module+el8.6.0+1 436 kB/s | 149 kB 00:00 2023-07-14T17:31:21.679 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-07-14T17:31:21.770 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================================ 2023-07-14T17:31:21.770 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2023-07-14T17:31:21.770 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================================ 2023-07-14T17:31:21.771 INFO:teuthology.orchestra.run.smithi107.stdout:Installing group/module packages: 2023-07-14T17:31:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T17:31:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T17:31:21.771 INFO:teuthology.orchestra.run.smithi107.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T17:31:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T17:31:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T17:31:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T17:31:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T17:31:21.772 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2023-07-14T17:31:21.772 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T17:31:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T17:31:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T17:31:21.773 INFO:teuthology.orchestra.run.smithi107.stdout:Installing module profiles: 2023-07-14T17:31:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: container-tools/common 2023-07-14T17:31:21.773 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-07-14T17:31:21.773 INFO:teuthology.orchestra.run.smithi107.stdout:Enabling module streams: 2023-07-14T17:31:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: container-tools rhel8 2023-07-14T17:31:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-07-14T17:31:21.774 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-07-14T17:31:21.774 INFO:teuthology.orchestra.run.smithi107.stdout:============================================================================================================ 2023-07-14T17:31:21.774 INFO:teuthology.orchestra.run.smithi107.stdout:Install 10 Packages 2023-07-14T17:31:21.774 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-07-14T17:31:21.775 INFO:teuthology.orchestra.run.smithi088.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 23 MB/s | 7.0 MB 00:00 2023-07-14T17:31:21.776 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 18 M 2023-07-14T17:31:21.776 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 65 M 2023-07-14T17:31:21.776 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-07-14T17:31:21.830 INFO:teuthology.orchestra.run.smithi088.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 11 MB/s | 2.2 MB 00:00 2023-07-14T17:31:21.831 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-07-14T17:31:21.831 INFO:teuthology.orchestra.run.smithi088.stdout:Total 22 MB/s | 18 MB 00:00 2023-07-14T17:31:21.968 INFO:teuthology.orchestra.run.smithi107.stdout:(1/10): yajl-2.1.0-10.el8.x86_64.rpm 209 kB/s | 41 kB 00:00 2023-07-14T17:31:22.000 INFO:teuthology.orchestra.run.smithi107.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 109 kB/s | 25 kB 00:00 2023-07-14T17:31:22.018 INFO:teuthology.orchestra.run.smithi107.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 383 kB/s | 94 kB 00:00 2023-07-14T17:31:22.029 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-07-14T17:31:22.047 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-07-14T17:31:22.048 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-07-14T17:31:22.101 INFO:teuthology.orchestra.run.smithi107.stdout:(4/10): cockpit-podman-49.1-1.module+el8.6.0+16 5.1 MB/s | 524 kB 00:00 2023-07-14T17:31:22.126 INFO:teuthology.orchestra.run.smithi107.stdout:(5/10): udica-0.2.6-4.module+el8.6.0+16771+28df 313 kB/s | 49 kB 00:00 2023-07-14T17:31:22.228 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-07-14T17:31:22.232 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-07-14T17:31:22.251 INFO:teuthology.orchestra.run.smithi107.stdout:(6/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.7 MB/s | 213 kB 00:00 2023-07-14T17:31:22.277 INFO:teuthology.orchestra.run.smithi107.stdout:(7/10): python3-podman-4.0.1-1.module+el8.6.0+1 852 kB/s | 149 kB 00:00 2023-07-14T17:31:22.358 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T17:31:22.427 INFO:teuthology.orchestra.run.smithi107.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 19 MB/s | 8.0 MB 00:00 2023-07-14T17:31:22.494 INFO:teuthology.orchestra.run.smithi107.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 10 MB/s | 2.2 MB 00:00 2023-07-14T17:31:22.611 INFO:teuthology.orchestra.run.smithi107.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 20 MB/s | 7.0 MB 00:00 2023-07-14T17:31:22.612 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-07-14T17:31:22.612 INFO:teuthology.orchestra.run.smithi107.stdout:Total 22 MB/s | 18 MB 00:00 2023-07-14T17:31:22.726 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-07-14T17:31:22.810 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-07-14T17:31:22.829 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-07-14T17:31:22.829 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-07-14T17:31:22.921 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T17:31:23.011 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-07-14T17:31:23.015 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-07-14T17:31:23.097 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T17:31:23.270 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T17:31:23.291 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T17:31:23.484 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T17:31:23.522 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T17:31:23.540 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-07-14T17:31:23.723 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T17:31:23.733 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T17:31:23.859 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T17:31:23.882 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T17:31:23.935 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T17:31:24.084 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T17:31:24.190 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T17:31:24.190 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T17:31:24.191 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T17:31:24.191 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T17:31:24.191 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T17:31:24.191 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T17:31:24.191 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T17:31:24.191 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T17:31:24.191 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T17:31:24.192 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T17:31:24.337 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T17:31:24.641 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T17:31:24.677 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T17:31:25.431 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T17:31:25.438 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T17:31:25.438 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:01:57 ago on Fri 14 Jul 2023 05:29:27 PM UTC. 2023-07-14T17:31:25.539 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T17:31:25.779 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T17:31:25.945 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T17:31:26.139 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T17:31:26.332 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T17:31:26.403 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T17:31:26.403 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T17:31:26.403 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T17:31:26.404 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T17:31:26.404 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T17:31:26.404 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T17:31:26.405 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T17:31:26.405 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T17:31:26.405 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T17:31:26.405 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T17:31:26.573 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T17:31:26.764 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T17:31:26.949 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T17:31:27.205 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T17:31:27.205 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T17:31:27.205 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T17:31:27.205 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T17:31:27.206 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T17:31:27.206 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T17:31:27.206 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T17:31:27.206 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T17:31:27.206 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T17:31:27.206 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T17:31:27.585 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T17:31:27.585 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:04:13 ago on Fri 14 Jul 2023 05:27:14 PM UTC. 2023-07-14T17:31:28.365 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T17:31:28.365 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:03:53 ago on Fri 14 Jul 2023 05:27:34 PM UTC. 2023-07-14T17:31:33.891 INFO:teuthology.orchestra.run.smithi178.stdout:Installed products updated. 2023-07-14T17:31:34.107 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-07-14T17:31:34.107 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-07-14T17:31:34.107 INFO:teuthology.orchestra.run.smithi178.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:31:34.108 INFO:teuthology.orchestra.run.smithi178.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T17:31:34.108 INFO:teuthology.orchestra.run.smithi178.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:31:34.108 INFO:teuthology.orchestra.run.smithi178.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T17:31:34.108 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T17:31:34.108 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T17:31:34.108 INFO:teuthology.orchestra.run.smithi178.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:31:34.108 INFO:teuthology.orchestra.run.smithi178.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:31:34.109 INFO:teuthology.orchestra.run.smithi178.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T17:31:34.109 INFO:teuthology.orchestra.run.smithi178.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T17:31:34.109 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-07-14T17:31:34.109 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-07-14T17:31:34.264 DEBUG:teuthology.parallel:result is None 2023-07-14T17:31:38.098 INFO:teuthology.orchestra.run.smithi088.stdout:Installed products updated. 2023-07-14T17:31:38.345 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:31:38.346 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-07-14T17:31:38.346 INFO:teuthology.orchestra.run.smithi088.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:31:38.346 INFO:teuthology.orchestra.run.smithi088.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T17:31:38.346 INFO:teuthology.orchestra.run.smithi088.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:31:38.346 INFO:teuthology.orchestra.run.smithi088.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T17:31:38.346 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T17:31:38.346 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T17:31:38.347 INFO:teuthology.orchestra.run.smithi088.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:31:38.347 INFO:teuthology.orchestra.run.smithi088.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:31:38.347 INFO:teuthology.orchestra.run.smithi088.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T17:31:38.347 INFO:teuthology.orchestra.run.smithi088.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T17:31:38.347 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:31:38.347 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-07-14T17:31:38.511 DEBUG:teuthology.parallel:result is None 2023-07-14T17:31:39.723 INFO:teuthology.orchestra.run.smithi107.stdout:Installed products updated. 2023-07-14T17:31:39.944 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-07-14T17:31:39.944 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-07-14T17:31:39.944 INFO:teuthology.orchestra.run.smithi107.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:31:39.945 INFO:teuthology.orchestra.run.smithi107.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T17:31:39.945 INFO:teuthology.orchestra.run.smithi107.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:31:39.945 INFO:teuthology.orchestra.run.smithi107.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T17:31:39.945 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T17:31:39.945 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T17:31:39.945 INFO:teuthology.orchestra.run.smithi107.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:31:39.945 INFO:teuthology.orchestra.run.smithi107.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T17:31:39.946 INFO:teuthology.orchestra.run.smithi107.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T17:31:39.946 INFO:teuthology.orchestra.run.smithi107.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T17:31:39.946 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-07-14T17:31:39.946 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-07-14T17:31:40.155 DEBUG:teuthology.parallel:result is None 2023-07-14T17:31:40.156 INFO:teuthology.run_tasks:Running task install... 2023-07-14T17:31:40.165 DEBUG:teuthology.task.install:project ceph 2023-07-14T17:31:40.166 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec'}} 2023-07-14T17:31:40.166 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec'} 2023-07-14T17:31:40.166 INFO:teuthology.task.install:Using flavor: default 2023-07-14T17:31:40.172 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-14T17:31:40.173 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-07-14T17:31:40.173 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T17:31:40.175 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T17:31:40.177 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T17:31:40.177 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T17:31:40.179 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T17:31:40.179 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T17:31:40.526 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T17:31:40.527 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T17:31:40.537 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T17:31:40.537 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T17:31:40.541 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T17:31:40.542 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T17:31:40.785 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T17:31:40.786 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:31:40.786 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T17:31:40.789 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T17:31:40.790 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-07-14T17:31:40.790 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T17:31:40.799 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T17:31:40.800 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-07-14T17:31:40.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T17:31:40.817 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T17:31:40.817 DEBUG:teuthology.orchestra.run.smithi088:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T17:31:40.824 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T17:31:40.824 DEBUG:teuthology.orchestra.run.smithi178:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T17:31:40.826 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T17:31:40.826 DEBUG:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T17:31:40.885 DEBUG:teuthology.orchestra.run.smithi088:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T17:31:40.894 DEBUG:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T17:31:40.896 DEBUG:teuthology.orchestra.run.smithi178:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T17:31:40.962 DEBUG:teuthology.orchestra.run.smithi088:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T17:31:40.970 DEBUG:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T17:31:40.976 DEBUG:teuthology.orchestra.run.smithi178:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T17:31:41.028 INFO:teuthology.orchestra.run.smithi088.stdout:check_obsoletes = 1 2023-07-14T17:31:41.029 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum clean all 2023-07-14T17:31:41.036 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2023-07-14T17:31:41.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum clean all 2023-07-14T17:31:41.044 INFO:teuthology.orchestra.run.smithi178.stdout:check_obsoletes = 1 2023-07-14T17:31:41.045 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum clean all 2023-07-14T17:31:41.400 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-07-14T17:31:41.421 INFO:teuthology.orchestra.run.smithi178.stdout:Updating Subscription Management repositories. 2023-07-14T17:31:41.423 INFO:teuthology.orchestra.run.smithi107.stdout:Updating Subscription Management repositories. 2023-07-14T17:31:42.446 INFO:teuthology.orchestra.run.smithi178.stdout:44 files removed 2023-07-14T17:31:42.465 INFO:teuthology.orchestra.run.smithi088.stdout:44 files removed 2023-07-14T17:31:42.476 INFO:teuthology.orchestra.run.smithi107.stdout:44 files removed 2023-07-14T17:31:42.500 DEBUG:teuthology.orchestra.run.smithi178:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T17:31:42.517 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T17:31:42.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T17:31:42.833 INFO:teuthology.orchestra.run.smithi178.stdout:Updating Subscription Management repositories. 2023-07-14T17:31:42.849 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-07-14T17:31:42.863 INFO:teuthology.orchestra.run.smithi107.stdout:Updating Subscription Management repositories. 2023-07-14T17:31:44.898 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 99 kB/s | 81 kB 00:00 2023-07-14T17:31:44.951 INFO:teuthology.orchestra.run.smithi178.stdout:ceph packages for x86_64 100 kB/s | 81 kB 00:00 2023-07-14T17:31:45.076 INFO:teuthology.orchestra.run.smithi088.stdout:ceph packages for x86_64 87 kB/s | 81 kB 00:00 2023-07-14T17:31:45.803 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 11 kB/s | 9.4 kB 00:00 2023-07-14T17:31:45.836 INFO:teuthology.orchestra.run.smithi178.stdout:ceph noarch packages 11 kB/s | 9.4 kB 00:00 2023-07-14T17:31:45.968 INFO:teuthology.orchestra.run.smithi088.stdout:ceph noarch packages 11 kB/s | 9.4 kB 00:00 2023-07-14T17:31:46.825 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 1.5 kB/s | 1.5 kB 00:01 2023-07-14T17:31:46.887 INFO:teuthology.orchestra.run.smithi178.stdout:ceph source packages 1.5 kB/s | 1.5 kB 00:01 2023-07-14T17:31:47.010 INFO:teuthology.orchestra.run.smithi088.stdout:ceph source packages 1.5 kB/s | 1.5 kB 00:01 2023-07-14T17:31:47.442 INFO:teuthology.orchestra.run.smithi107.stdout:Copr repo for python3-asyncssh owned by ceph 5.8 kB/s | 3.5 kB 00:00 2023-07-14T17:31:47.528 INFO:teuthology.orchestra.run.smithi178.stdout:Copr repo for python3-asyncssh owned by ceph 5.6 kB/s | 3.5 kB 00:00 2023-07-14T17:31:47.644 INFO:teuthology.orchestra.run.smithi088.stdout:Copr repo for python3-asyncssh owned by ceph 5.7 kB/s | 3.5 kB 00:00 2023-07-14T17:31:50.187 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 6.2 MB/s | 16 MB 00:02 2023-07-14T17:31:50.728 INFO:teuthology.orchestra.run.smithi178.stdout:Extra Packages for Enterprise Linux 4.9 MB/s | 16 MB 00:03 2023-07-14T17:31:53.686 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 277 kB/s | 24 kB 00:00 2023-07-14T17:31:54.154 INFO:teuthology.orchestra.run.smithi178.stdout:lab-extras 484 kB/s | 24 kB 00:00 2023-07-14T17:31:54.813 INFO:teuthology.orchestra.run.smithi088.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 45 MB/s | 47 MB 00:01 2023-07-14T17:31:55.308 INFO:teuthology.orchestra.run.smithi178.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 44 MB/s | 47 MB 00:01 2023-07-14T17:32:03.190 INFO:teuthology.orchestra.run.smithi088.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 43 MB/s | 53 MB 00:01 2023-07-14T17:32:03.532 INFO:teuthology.orchestra.run.smithi178.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 45 MB/s | 53 MB 00:01 2023-07-14T17:32:06.799 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 826 kB/s | 16 MB 00:19 2023-07-14T17:32:10.223 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 527 kB/s | 24 kB 00:00 2023-07-14T17:32:11.508 INFO:teuthology.orchestra.run.smithi107.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 44 MB/s | 53 MB 00:01 2023-07-14T17:32:14.423 INFO:teuthology.orchestra.run.smithi088.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T17:32:14.424 INFO:teuthology.orchestra.run.smithi088.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T17:32:14.438 INFO:teuthology.orchestra.run.smithi088.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T17:32:14.439 INFO:teuthology.orchestra.run.smithi088.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T17:32:14.440 INFO:teuthology.orchestra.run.smithi088.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T17:32:14.442 INFO:teuthology.orchestra.run.smithi088.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T17:32:14.444 INFO:teuthology.orchestra.run.smithi088.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T17:32:14.447 INFO:teuthology.orchestra.run.smithi088.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T17:32:14.448 INFO:teuthology.orchestra.run.smithi088.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T17:32:14.582 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-07-14T17:32:14.595 INFO:teuthology.orchestra.run.smithi088.stdout:=========================================================================================================================== 2023-07-14T17:32:14.595 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2023-07-14T17:32:14.595 INFO:teuthology.orchestra.run.smithi088.stdout:=========================================================================================================================== 2023-07-14T17:32:14.595 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-07-14T17:32:14.595 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T17:32:14.595 INFO:teuthology.orchestra.run.smithi088.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T17:32:14.596 INFO:teuthology.orchestra.run.smithi088.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T17:32:14.596 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T17:32:14.596 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T17:32:14.596 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T17:32:14.596 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T17:32:14.596 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T17:32:14.596 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T17:32:14.597 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T17:32:14.597 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T17:32:14.597 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T17:32:14.597 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T17:32:14.598 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T17:32:14.598 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T17:32:14.598 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T17:32:14.598 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T17:32:14.598 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T17:32:14.598 INFO:teuthology.orchestra.run.smithi088.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T17:32:14.598 INFO:teuthology.orchestra.run.smithi088.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T17:32:14.599 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T17:32:14.599 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T17:32:14.599 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T17:32:14.599 INFO:teuthology.orchestra.run.smithi088.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T17:32:14.599 INFO:teuthology.orchestra.run.smithi088.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T17:32:14.599 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T17:32:14.599 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T17:32:14.600 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T17:32:14.600 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T17:32:14.600 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T17:32:14.600 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T17:32:14.600 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T17:32:14.600 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T17:32:14.600 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T17:32:14.601 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T17:32:14.601 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T17:32:14.601 INFO:teuthology.orchestra.run.smithi088.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T17:32:14.601 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrading: 2023-07-14T17:32:14.601 INFO:teuthology.orchestra.run.smithi088.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T17:32:14.601 INFO:teuthology.orchestra.run.smithi088.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T17:32:14.601 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T17:32:14.601 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2023-07-14T17:32:14.602 INFO:teuthology.orchestra.run.smithi088.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T17:32:14.602 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T17:32:14.602 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T17:32:14.602 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T17:32:14.602 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T17:32:14.602 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T17:32:14.602 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T17:32:14.603 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T17:32:14.603 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T17:32:14.603 INFO:teuthology.orchestra.run.smithi088.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T17:32:14.603 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T17:32:14.603 INFO:teuthology.orchestra.run.smithi088.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T17:32:14.603 INFO:teuthology.orchestra.run.smithi088.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T17:32:14.603 INFO:teuthology.orchestra.run.smithi088.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T17:32:14.603 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T17:32:14.604 INFO:teuthology.orchestra.run.smithi088.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T17:32:14.604 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T17:32:14.605 INFO:teuthology.orchestra.run.smithi088.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T17:32:14.606 INFO:teuthology.orchestra.run.smithi088.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T17:32:14.606 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T17:32:14.606 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T17:32:14.606 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T17:32:14.606 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T17:32:14.607 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T17:32:14.607 INFO:teuthology.orchestra.run.smithi088.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T17:32:14.607 INFO:teuthology.orchestra.run.smithi088.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T17:32:14.607 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T17:32:14.607 INFO:teuthology.orchestra.run.smithi088.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T17:32:14.607 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T17:32:14.607 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T17:32:14.607 INFO:teuthology.orchestra.run.smithi088.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T17:32:14.608 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T17:32:14.608 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T17:32:14.608 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T17:32:14.608 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T17:32:14.608 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T17:32:14.608 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T17:32:14.608 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T17:32:14.609 INFO:teuthology.orchestra.run.smithi088.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T17:32:14.609 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T17:32:14.609 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T17:32:14.609 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T17:32:14.609 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T17:32:14.609 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T17:32:14.609 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T17:32:14.610 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T17:32:14.610 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T17:32:14.610 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T17:32:14.610 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T17:32:14.610 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T17:32:14.610 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T17:32:14.610 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T17:32:14.611 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T17:32:14.611 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T17:32:14.611 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T17:32:14.611 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T17:32:14.611 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T17:32:14.611 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T17:32:14.611 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T17:32:14.611 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T17:32:14.612 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T17:32:14.612 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T17:32:14.612 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T17:32:14.612 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T17:32:14.612 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T17:32:14.612 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T17:32:14.612 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T17:32:14.613 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T17:32:14.613 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T17:32:14.613 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T17:32:14.613 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T17:32:14.613 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T17:32:14.613 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T17:32:14.613 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T17:32:14.614 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T17:32:14.614 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T17:32:14.614 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T17:32:14.614 INFO:teuthology.orchestra.run.smithi088.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T17:32:14.614 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T17:32:14.614 INFO:teuthology.orchestra.run.smithi088.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T17:32:14.614 INFO:teuthology.orchestra.run.smithi088.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T17:32:14.615 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T17:32:14.615 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T17:32:14.615 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T17:32:14.615 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T17:32:14.615 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T17:32:14.615 INFO:teuthology.orchestra.run.smithi088.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T17:32:14.615 INFO:teuthology.orchestra.run.smithi088.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T17:32:14.616 INFO:teuthology.orchestra.run.smithi088.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T17:32:14.616 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T17:32:14.616 INFO:teuthology.orchestra.run.smithi088.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T17:32:14.616 INFO:teuthology.orchestra.run.smithi088.stdout:Enabling module streams: 2023-07-14T17:32:14.616 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql 10 2023-07-14T17:32:14.616 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:32:14.616 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-07-14T17:32:14.617 INFO:teuthology.orchestra.run.smithi088.stdout:=========================================================================================================================== 2023-07-14T17:32:14.617 INFO:teuthology.orchestra.run.smithi088.stdout:Install 128 Packages 2023-07-14T17:32:14.617 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrade 3 Packages 2023-07-14T17:32:14.617 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:32:14.618 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 206 M 2023-07-14T17:32:14.618 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-07-14T17:32:14.622 INFO:teuthology.orchestra.run.smithi178.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T17:32:14.623 INFO:teuthology.orchestra.run.smithi178.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T17:32:14.638 INFO:teuthology.orchestra.run.smithi178.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T17:32:14.639 INFO:teuthology.orchestra.run.smithi178.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T17:32:14.640 INFO:teuthology.orchestra.run.smithi178.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T17:32:14.642 INFO:teuthology.orchestra.run.smithi178.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T17:32:14.644 INFO:teuthology.orchestra.run.smithi178.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T17:32:14.647 INFO:teuthology.orchestra.run.smithi178.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T17:32:14.648 INFO:teuthology.orchestra.run.smithi178.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T17:32:14.780 INFO:teuthology.orchestra.run.smithi178.stdout:Dependencies resolved. 2023-07-14T17:32:14.792 INFO:teuthology.orchestra.run.smithi178.stdout:=========================================================================================================================== 2023-07-14T17:32:14.792 INFO:teuthology.orchestra.run.smithi178.stdout: Package Arch Version Repository Size 2023-07-14T17:32:14.792 INFO:teuthology.orchestra.run.smithi178.stdout:=========================================================================================================================== 2023-07-14T17:32:14.792 INFO:teuthology.orchestra.run.smithi178.stdout:Installing: 2023-07-14T17:32:14.793 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T17:32:14.793 INFO:teuthology.orchestra.run.smithi178.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T17:32:14.793 INFO:teuthology.orchestra.run.smithi178.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T17:32:14.793 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T17:32:14.793 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T17:32:14.793 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T17:32:14.794 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T17:32:14.794 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T17:32:14.794 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T17:32:14.794 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T17:32:14.794 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T17:32:14.794 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T17:32:14.795 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T17:32:14.795 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T17:32:14.795 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T17:32:14.795 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T17:32:14.796 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T17:32:14.796 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T17:32:14.796 INFO:teuthology.orchestra.run.smithi178.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T17:32:14.796 INFO:teuthology.orchestra.run.smithi178.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T17:32:14.796 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T17:32:14.796 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T17:32:14.797 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T17:32:14.797 INFO:teuthology.orchestra.run.smithi178.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T17:32:14.797 INFO:teuthology.orchestra.run.smithi178.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T17:32:14.797 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T17:32:14.799 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T17:32:14.799 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T17:32:14.800 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T17:32:14.800 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T17:32:14.800 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T17:32:14.800 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T17:32:14.800 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T17:32:14.800 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T17:32:14.800 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T17:32:14.801 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T17:32:14.801 INFO:teuthology.orchestra.run.smithi178.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T17:32:14.801 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrading: 2023-07-14T17:32:14.801 INFO:teuthology.orchestra.run.smithi178.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T17:32:14.801 INFO:teuthology.orchestra.run.smithi178.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T17:32:14.801 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T17:32:14.801 INFO:teuthology.orchestra.run.smithi178.stdout:Installing dependencies: 2023-07-14T17:32:14.802 INFO:teuthology.orchestra.run.smithi178.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T17:32:14.802 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T17:32:14.802 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T17:32:14.802 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T17:32:14.802 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T17:32:14.802 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T17:32:14.802 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T17:32:14.803 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T17:32:14.803 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T17:32:14.803 INFO:teuthology.orchestra.run.smithi178.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T17:32:14.803 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T17:32:14.803 INFO:teuthology.orchestra.run.smithi178.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T17:32:14.803 INFO:teuthology.orchestra.run.smithi178.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T17:32:14.804 INFO:teuthology.orchestra.run.smithi178.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T17:32:14.804 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T17:32:14.804 INFO:teuthology.orchestra.run.smithi178.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T17:32:14.804 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T17:32:14.804 INFO:teuthology.orchestra.run.smithi178.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T17:32:14.804 INFO:teuthology.orchestra.run.smithi178.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T17:32:14.804 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T17:32:14.805 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T17:32:14.805 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T17:32:14.805 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T17:32:14.805 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T17:32:14.806 INFO:teuthology.orchestra.run.smithi178.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T17:32:14.806 INFO:teuthology.orchestra.run.smithi178.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T17:32:14.806 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T17:32:14.806 INFO:teuthology.orchestra.run.smithi178.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T17:32:14.806 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T17:32:14.806 INFO:teuthology.orchestra.run.smithi178.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T17:32:14.806 INFO:teuthology.orchestra.run.smithi178.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T17:32:14.807 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T17:32:14.807 INFO:teuthology.orchestra.run.smithi178.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T17:32:14.807 INFO:teuthology.orchestra.run.smithi178.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T17:32:14.807 INFO:teuthology.orchestra.run.smithi178.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T17:32:14.807 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T17:32:14.807 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T17:32:14.807 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T17:32:14.808 INFO:teuthology.orchestra.run.smithi178.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T17:32:14.808 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T17:32:14.808 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T17:32:14.808 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T17:32:14.808 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T17:32:14.808 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T17:32:14.808 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T17:32:14.809 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T17:32:14.809 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T17:32:14.809 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T17:32:14.809 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T17:32:14.809 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T17:32:14.809 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T17:32:14.809 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T17:32:14.810 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T17:32:14.810 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T17:32:14.810 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T17:32:14.810 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T17:32:14.810 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T17:32:14.810 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T17:32:14.810 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T17:32:14.811 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T17:32:14.811 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T17:32:14.811 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T17:32:14.811 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T17:32:14.811 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T17:32:14.811 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T17:32:14.811 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T17:32:14.812 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T17:32:14.812 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T17:32:14.812 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T17:32:14.812 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T17:32:14.812 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T17:32:14.812 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T17:32:14.812 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T17:32:14.813 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T17:32:14.813 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T17:32:14.813 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T17:32:14.813 INFO:teuthology.orchestra.run.smithi178.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T17:32:14.813 INFO:teuthology.orchestra.run.smithi178.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T17:32:14.813 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T17:32:14.813 INFO:teuthology.orchestra.run.smithi178.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T17:32:14.814 INFO:teuthology.orchestra.run.smithi178.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T17:32:14.814 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T17:32:14.814 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T17:32:14.814 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T17:32:14.814 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T17:32:14.814 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T17:32:14.814 INFO:teuthology.orchestra.run.smithi178.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T17:32:14.814 INFO:teuthology.orchestra.run.smithi178.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T17:32:14.815 INFO:teuthology.orchestra.run.smithi178.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T17:32:14.815 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T17:32:14.815 INFO:teuthology.orchestra.run.smithi178.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T17:32:14.815 INFO:teuthology.orchestra.run.smithi178.stdout:Enabling module streams: 2023-07-14T17:32:14.815 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql 10 2023-07-14T17:32:14.815 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-07-14T17:32:14.815 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction Summary 2023-07-14T17:32:14.816 INFO:teuthology.orchestra.run.smithi178.stdout:=========================================================================================================================== 2023-07-14T17:32:14.816 INFO:teuthology.orchestra.run.smithi178.stdout:Install 128 Packages 2023-07-14T17:32:14.816 INFO:teuthology.orchestra.run.smithi178.stdout:Upgrade 3 Packages 2023-07-14T17:32:14.816 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-07-14T17:32:14.817 INFO:teuthology.orchestra.run.smithi178.stdout:Total download size: 206 M 2023-07-14T17:32:14.817 INFO:teuthology.orchestra.run.smithi178.stdout:Downloading Packages: 2023-07-14T17:32:15.221 INFO:teuthology.orchestra.run.smithi088.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 18 kB/s | 6.4 kB 00:00 2023-07-14T17:32:15.432 INFO:teuthology.orchestra.run.smithi178.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 16 kB/s | 6.4 kB 00:00 2023-07-14T17:32:15.855 INFO:teuthology.orchestra.run.smithi088.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 1.3 MB/s | 850 kB 00:00 2023-07-14T17:32:15.980 INFO:teuthology.orchestra.run.smithi088.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 1.3 MB/s | 167 kB 00:00 2023-07-14T17:32:16.090 INFO:teuthology.orchestra.run.smithi178.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 1.3 MB/s | 850 kB 00:00 2023-07-14T17:32:16.240 INFO:teuthology.orchestra.run.smithi178.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 1.1 MB/s | 167 kB 00:00 2023-07-14T17:32:16.289 INFO:teuthology.orchestra.run.smithi088.stdout:(4/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 3.8 MB/s | 5.4 MB 00:01 2023-07-14T17:32:16.706 INFO:teuthology.orchestra.run.smithi088.stdout:(5/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 3.7 MB/s | 1.6 MB 00:00 2023-07-14T17:32:16.741 INFO:teuthology.orchestra.run.smithi178.stdout:(4/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 4.6 MB/s | 2.3 MB 00:00 2023-07-14T17:32:17.099 INFO:teuthology.orchestra.run.smithi178.stdout:(5/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 4.3 MB/s | 1.6 MB 00:00 2023-07-14T17:32:17.123 INFO:teuthology.orchestra.run.smithi088.stdout:(6/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 2.0 MB/s | 2.3 MB 00:01 2023-07-14T17:32:17.456 INFO:teuthology.orchestra.run.smithi088.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 6.2 MB/s | 4.6 MB 00:00 2023-07-14T17:32:17.708 INFO:teuthology.orchestra.run.smithi178.stdout:(6/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 2.0 MB/s | 5.4 MB 00:02 2023-07-14T17:32:17.958 INFO:teuthology.orchestra.run.smithi178.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 5.4 MB/s | 4.6 MB 00:00 2023-07-14T17:32:18.565 INFO:teuthology.orchestra.run.smithi088.stdout:(8/131): ceph-common-18.0.0-4940.g81c9387d.el8. 6.0 MB/s | 22 MB 00:03 2023-07-14T17:32:18.665 INFO:teuthology.orchestra.run.smithi088.stdout:(9/131): ceph-selinux-18.0.0-4940.g81c9387d.el8 243 kB/s | 24 kB 00:00 2023-07-14T17:32:19.852 INFO:teuthology.orchestra.run.smithi107.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 44 MB/s | 47 MB 00:01 2023-07-14T17:32:19.907 INFO:teuthology.orchestra.run.smithi088.stdout:(10/131): ceph-radosgw-18.0.0-4940.g81c9387d.el 5.5 MB/s | 14 MB 00:02 2023-07-14T17:32:20.057 INFO:teuthology.orchestra.run.smithi088.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 1.7 MB/s | 260 kB 00:00 2023-07-14T17:32:20.190 INFO:teuthology.orchestra.run.smithi088.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 233 kB/s | 31 kB 00:00 2023-07-14T17:32:20.450 INFO:teuthology.orchestra.run.smithi088.stdout:(13/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 2.9 MB/s | 772 kB 00:00 2023-07-14T17:32:20.574 INFO:teuthology.orchestra.run.smithi088.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 1.1 MB/s | 135 kB 00:00 2023-07-14T17:32:20.708 INFO:teuthology.orchestra.run.smithi088.stdout:(15/131): librados-devel-18.0.0-4940.g81c9387d. 1.0 MB/s | 139 kB 00:00 2023-07-14T17:32:20.850 INFO:teuthology.orchestra.run.smithi088.stdout:(16/131): libradosstriper1-18.0.0-4940.g81c9387 3.3 MB/s | 473 kB 00:00 2023-07-14T17:32:21.033 INFO:teuthology.orchestra.run.smithi178.stdout:(8/131): ceph-radosgw-18.0.0-4940.g81c9387d.el8 4.4 MB/s | 14 MB 00:03 2023-07-14T17:32:21.275 INFO:teuthology.orchestra.run.smithi088.stdout:(17/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x8 4.3 MB/s | 18 MB 00:04 2023-07-14T17:32:21.308 INFO:teuthology.orchestra.run.smithi178.stdout:(9/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x86 4.9 MB/s | 18 MB 00:03 2023-07-14T17:32:21.371 INFO:teuthology.orchestra.run.smithi178.stdout:(10/131): ceph-selinux-18.0.0-4940.g81c9387d.el 72 kB/s | 24 kB 00:00 2023-07-14T17:32:21.409 INFO:teuthology.orchestra.run.smithi088.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 336 kB/s | 45 kB 00:00 2023-07-14T17:32:21.551 INFO:teuthology.orchestra.run.smithi088.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 865 kB/s | 122 kB 00:00 2023-07-14T17:32:21.675 INFO:teuthology.orchestra.run.smithi178.stdout:(11/131): ceph-common-18.0.0-4940.g81c9387d.el8 3.3 MB/s | 22 MB 00:06 2023-07-14T17:32:21.693 INFO:teuthology.orchestra.run.smithi088.stdout:(20/131): python3-cephfs-18.0.0-4940.g81c9387d. 1.4 MB/s | 197 kB 00:00 2023-07-14T17:32:21.701 INFO:teuthology.orchestra.run.smithi178.stdout:(12/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 790 kB/s | 260 kB 00:00 2023-07-14T17:32:21.818 INFO:teuthology.orchestra.run.smithi178.stdout:(13/131): libcephfs-devel-18.0.0-4940.g81c9387d 218 kB/s | 31 kB 00:00 2023-07-14T17:32:21.843 INFO:teuthology.orchestra.run.smithi088.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 2.4 MB/s | 371 kB 00:00 2023-07-14T17:32:21.968 INFO:teuthology.orchestra.run.smithi178.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 904 kB/s | 135 kB 00:00 2023-07-14T17:32:22.018 INFO:teuthology.orchestra.run.smithi088.stdout:(22/131): python3-rbd-18.0.0-4940.g81c9387d.el8 2.0 MB/s | 364 kB 00:00 2023-07-14T17:32:22.027 INFO:teuthology.orchestra.run.smithi178.stdout:(15/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 2.3 MB/s | 772 kB 00:00 2023-07-14T17:32:22.127 INFO:teuthology.orchestra.run.smithi178.stdout:(16/131): librados-devel-18.0.0-4940.g81c9387d. 877 kB/s | 139 kB 00:00 2023-07-14T17:32:22.168 INFO:teuthology.orchestra.run.smithi088.stdout:(23/131): python3-rgw-18.0.0-4940.g81c9387d.el8 711 kB/s | 107 kB 00:00 2023-07-14T17:32:22.202 INFO:teuthology.orchestra.run.smithi178.stdout:(17/131): libradosstriper1-18.0.0-4940.g81c9387 2.6 MB/s | 473 kB 00:00 2023-07-14T17:32:22.310 INFO:teuthology.orchestra.run.smithi088.stdout:(24/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 660 kB/s | 94 kB 00:00 2023-07-14T17:32:22.369 INFO:teuthology.orchestra.run.smithi178.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 269 kB/s | 45 kB 00:00 2023-07-14T17:32:22.444 INFO:teuthology.orchestra.run.smithi088.stdout:(25/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 4.3 MB/s | 6.8 MB 00:01 2023-07-14T17:32:22.603 INFO:teuthology.orchestra.run.smithi178.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 525 kB/s | 122 kB 00:00 2023-07-14T17:32:22.636 INFO:teuthology.orchestra.run.smithi088.stdout:(26/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 934 kB/s | 179 kB 00:00 2023-07-14T17:32:22.745 INFO:teuthology.orchestra.run.smithi178.stdout:(20/131): python3-cephfs-18.0.0-4940.g81c9387d. 1.4 MB/s | 197 kB 00:00 2023-07-14T17:32:22.753 INFO:teuthology.orchestra.run.smithi088.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4940.g 198 kB/s | 23 kB 00:00 2023-07-14T17:32:22.886 INFO:teuthology.orchestra.run.smithi178.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 2.6 MB/s | 371 kB 00:00 2023-07-14T17:32:22.895 INFO:teuthology.orchestra.run.smithi088.stdout:(28/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 912 kB/s | 129 kB 00:00 2023-07-14T17:32:23.037 INFO:teuthology.orchestra.run.smithi178.stdout:(22/131): python3-rbd-18.0.0-4940.g81c9387d.el8 2.4 MB/s | 364 kB 00:00 2023-07-14T17:32:23.162 INFO:teuthology.orchestra.run.smithi178.stdout:(23/131): python3-rgw-18.0.0-4940.g81c9387d.el8 853 kB/s | 107 kB 00:00 2023-07-14T17:32:23.295 INFO:teuthology.orchestra.run.smithi088.stdout:(29/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 4.3 MB/s | 1.7 MB 00:00 2023-07-14T17:32:23.321 INFO:teuthology.orchestra.run.smithi178.stdout:(24/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 592 kB/s | 94 kB 00:00 2023-07-14T17:32:23.354 INFO:teuthology.orchestra.run.smithi088.stdout:(30/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 3.4 MB/s | 3.5 MB 00:01 2023-07-14T17:32:23.512 INFO:teuthology.orchestra.run.smithi088.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 1.5 MB/s | 241 kB 00:00 2023-07-14T17:32:23.655 INFO:teuthology.orchestra.run.smithi088.stdout:(32/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 334 kB/s | 47 kB 00:00 2023-07-14T17:32:23.771 INFO:teuthology.orchestra.run.smithi088.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4940.g8 125 kB/s | 15 kB 00:00 2023-07-14T17:32:23.905 INFO:teuthology.orchestra.run.smithi088.stdout:(34/131): ceph-volume-18.0.0-4940.g81c9387d.el8 1.9 MB/s | 258 kB 00:00 2023-07-14T17:32:24.038 INFO:teuthology.orchestra.run.smithi088.stdout:(35/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 1.2 MB/s | 165 kB 00:00 2023-07-14T17:32:24.147 INFO:teuthology.orchestra.run.smithi088.stdout:(36/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 166 kB/s | 18 kB 00:00 2023-07-14T17:32:24.246 INFO:teuthology.orchestra.run.smithi178.stdout:(25/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 3.8 MB/s | 3.5 MB 00:00 2023-07-14T17:32:24.379 INFO:teuthology.orchestra.run.smithi178.stdout:(26/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 3.0 MB/s | 6.8 MB 00:02 2023-07-14T17:32:24.405 INFO:teuthology.orchestra.run.smithi178.stdout:(27/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 1.1 MB/s | 179 kB 00:00 2023-07-14T17:32:24.522 INFO:teuthology.orchestra.run.smithi178.stdout:(28/131): ceph-grafana-dashboards-18.0.0-4940.g 162 kB/s | 23 kB 00:00 2023-07-14T17:32:24.547 INFO:teuthology.orchestra.run.smithi178.stdout:(29/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 910 kB/s | 129 kB 00:00 2023-07-14T17:32:24.756 INFO:teuthology.orchestra.run.smithi088.stdout:(37/131): ceph-mgr-diskprediction-local-18.0.0- 5.1 MB/s | 7.4 MB 00:01 2023-07-14T17:32:24.781 INFO:teuthology.orchestra.run.smithi088.stdout:(38/131): gperftools-libs-2.7-9.el8.x86_64.rpm 483 kB/s | 306 kB 00:00 2023-07-14T17:32:24.898 INFO:teuthology.orchestra.run.smithi088.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 644 kB/s | 75 kB 00:00 2023-07-14T17:32:25.055 INFO:teuthology.orchestra.run.smithi178.stdout:(30/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 3.2 MB/s | 1.7 MB 00:00 2023-07-14T17:32:25.115 INFO:teuthology.orchestra.run.smithi088.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 2.1 MB/s | 476 kB 00:00 2023-07-14T17:32:25.140 INFO:teuthology.orchestra.run.smithi088.stdout:(41/131): liboath-2.6.2-3.el8.x86_64.rpm 153 kB/s | 59 kB 00:00 2023-07-14T17:32:25.172 INFO:teuthology.orchestra.run.smithi178.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 2.0 MB/s | 241 kB 00:00 2023-07-14T17:32:25.224 INFO:teuthology.orchestra.run.smithi088.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 402 kB/s | 44 kB 00:00 2023-07-14T17:32:25.298 INFO:teuthology.orchestra.run.smithi178.stdout:(32/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 379 kB/s | 47 kB 00:00 2023-07-14T17:32:25.332 INFO:teuthology.orchestra.run.smithi088.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 305 kB/s | 33 kB 00:00 2023-07-14T17:32:25.382 INFO:teuthology.orchestra.run.smithi088.stdout:(44/131): python3-beautifulsoup4-4.6.3-2.el8.1. 765 kB/s | 185 kB 00:00 2023-07-14T17:32:25.406 INFO:teuthology.orchestra.run.smithi178.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4940.g8 134 kB/s | 15 kB 00:00 2023-07-14T17:32:25.449 INFO:teuthology.orchestra.run.smithi088.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 139 kB/s | 16 kB 00:00 2023-07-14T17:32:25.541 INFO:teuthology.orchestra.run.smithi088.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.1 MB/s | 173 kB 00:00 2023-07-14T17:32:25.556 INFO:teuthology.orchestra.run.smithi178.stdout:(34/131): ceph-volume-18.0.0-4940.g81c9387d.el8 1.7 MB/s | 258 kB 00:00 2023-07-14T17:32:25.583 INFO:teuthology.orchestra.run.smithi088.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 2.8 MB/s | 384 kB 00:00 2023-07-14T17:32:25.667 INFO:teuthology.orchestra.run.smithi088.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 648 kB/s | 81 kB 00:00 2023-07-14T17:32:25.715 INFO:teuthology.orchestra.run.smithi178.stdout:(35/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 1.0 MB/s | 165 kB 00:00 2023-07-14T17:32:25.717 INFO:teuthology.orchestra.run.smithi088.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 83 kB/s | 11 kB 00:00 2023-07-14T17:32:25.792 INFO:teuthology.orchestra.run.smithi088.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 140 kB/s | 18 kB 00:00 2023-07-14T17:32:25.832 INFO:teuthology.orchestra.run.smithi178.stdout:(36/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 154 kB/s | 18 kB 00:00 2023-07-14T17:32:25.909 INFO:teuthology.orchestra.run.smithi088.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 418 kB/s | 49 kB 00:00 2023-07-14T17:32:25.960 INFO:teuthology.orchestra.run.smithi088.stdout:(52/131): python3-kubernetes-11.0.0-6.el8.noarc 4.8 MB/s | 1.2 MB 00:00 2023-07-14T17:32:26.043 INFO:teuthology.orchestra.run.smithi088.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 439 kB/s | 59 kB 00:00 2023-07-14T17:32:26.077 INFO:teuthology.orchestra.run.smithi088.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 515 kB/s | 60 kB 00:00 2023-07-14T17:32:26.194 INFO:teuthology.orchestra.run.smithi088.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 134 kB/s | 16 kB 00:00 2023-07-14T17:32:26.199 INFO:teuthology.orchestra.run.smithi178.stdout:(37/131): ceph-mgr-diskprediction-local-18.0.0- 4.5 MB/s | 7.4 MB 00:01 2023-07-14T17:32:26.314 INFO:teuthology.orchestra.run.smithi088.stdout:(56/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.0 MB/s | 283 kB 00:00 2023-07-14T17:32:26.524 INFO:teuthology.orchestra.run.smithi178.stdout:(38/131): gperftools-libs-2.7-9.el8.x86_64.rpm 442 kB/s | 306 kB 00:00 2023-07-14T17:32:26.545 INFO:teuthology.orchestra.run.smithi088.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 96 kB/s | 34 kB 00:00 2023-07-14T17:32:26.570 INFO:teuthology.orchestra.run.smithi088.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 924 kB/s | 196 kB 00:00 2023-07-14T17:32:26.666 INFO:teuthology.orchestra.run.smithi178.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 530 kB/s | 75 kB 00:00 2023-07-14T17:32:26.670 INFO:teuthology.orchestra.run.smithi088.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 491 kB/s | 61 kB 00:00 2023-07-14T17:32:26.692 INFO:teuthology.orchestra.run.smithi178.stdout:(40/131): liboath-2.6.2-3.el8.x86_64.rpm 119 kB/s | 59 kB 00:00 2023-07-14T17:32:26.713 INFO:teuthology.orchestra.run.smithi088.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 132 kB/s | 19 kB 00:00 2023-07-14T17:32:26.796 INFO:teuthology.orchestra.run.smithi088.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 191 kB/s | 24 kB 00:00 2023-07-14T17:32:26.842 INFO:teuthology.orchestra.run.smithi178.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 291 kB/s | 44 kB 00:00 2023-07-14T17:32:26.846 INFO:teuthology.orchestra.run.smithi088.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 219 kB/s | 29 kB 00:00 2023-07-14T17:32:26.897 INFO:teuthology.orchestra.run.smithi088.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 274 kB/s | 27 kB 00:00 2023-07-14T17:32:26.926 INFO:teuthology.orchestra.run.smithi178.stdout:(42/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.8 MB/s | 476 kB 00:00 2023-07-14T17:32:26.997 INFO:teuthology.orchestra.run.smithi088.stdout:(64/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.5 MB/s | 251 kB 00:00 2023-07-14T17:32:27.056 INFO:teuthology.orchestra.run.smithi088.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 1.2 MB/s | 255 kB 00:00 2023-07-14T17:32:27.085 INFO:teuthology.orchestra.run.smithi178.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 207 kB/s | 33 kB 00:00 2023-07-14T17:32:27.114 INFO:teuthology.orchestra.run.smithi088.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 524 kB/s | 61 kB 00:00 2023-07-14T17:32:27.118 INFO:teuthology.orchestra.run.smithi178.stdout:(44/131): python3-beautifulsoup4-4.6.3-2.el8.1. 672 kB/s | 185 kB 00:00 2023-07-14T17:32:27.198 INFO:teuthology.orchestra.run.smithi088.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 602 kB/s | 85 kB 00:00 2023-07-14T17:32:27.218 INFO:teuthology.orchestra.run.smithi178.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 123 kB/s | 16 kB 00:00 2023-07-14T17:32:27.239 INFO:teuthology.orchestra.run.smithi088.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 183 kB/s | 23 kB 00:00 2023-07-14T17:32:27.260 INFO:teuthology.orchestra.run.smithi178.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.2 MB/s | 173 kB 00:00 2023-07-14T17:32:27.273 INFO:teuthology.orchestra.run.smithi088.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.0 MB/s | 70 kB 00:00 2023-07-14T17:32:27.419 INFO:teuthology.orchestra.run.smithi178.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 513 kB/s | 81 kB 00:00 2023-07-14T17:32:27.486 INFO:teuthology.orchestra.run.smithi178.stdout:(48/131): python3-cherrypy-18.4.0-1.el8.noarch. 1.4 MB/s | 384 kB 00:00 2023-07-14T17:32:27.561 INFO:teuthology.orchestra.run.smithi178.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 78 kB/s | 11 kB 00:00 2023-07-14T17:32:27.637 INFO:teuthology.orchestra.run.smithi178.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 117 kB/s | 18 kB 00:00 2023-07-14T17:32:27.932 INFO:teuthology.orchestra.run.smithi088.stdout:(70/131): python3-scipy-1.0.0-21.module+el8.5.0 21 MB/s | 14 MB 00:00 2023-07-14T17:32:28.004 INFO:teuthology.orchestra.run.smithi178.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 133 kB/s | 49 kB 00:00 2023-07-14T17:32:28.107 INFO:teuthology.orchestra.run.smithi088.stdout:(71/131): thrift-0.13.0-2.el8.x86_64.rpm 1.9 MB/s | 1.7 MB 00:00 2023-07-14T17:32:28.162 INFO:teuthology.orchestra.run.smithi178.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 371 kB/s | 59 kB 00:00 2023-07-14T17:32:28.204 INFO:teuthology.orchestra.run.smithi178.stdout:(53/131): python3-kubernetes-11.0.0-6.el8.noarc 1.8 MB/s | 1.2 MB 00:00 2023-07-14T17:32:28.249 INFO:teuthology.orchestra.run.smithi088.stdout:(72/131): python3-pyasn1-modules-0.3.7-6.el8.no 348 kB/s | 110 kB 00:00 2023-07-14T17:32:28.338 INFO:teuthology.orchestra.run.smithi178.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 344 kB/s | 60 kB 00:00 2023-07-14T17:32:28.363 INFO:teuthology.orchestra.run.smithi178.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.7 MB/s | 283 kB 00:00 2023-07-14T17:32:28.505 INFO:teuthology.orchestra.run.smithi178.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 93 kB/s | 16 kB 00:00 2023-07-14T17:32:28.524 INFO:teuthology.orchestra.run.smithi088.stdout:(73/131): python3-pytz-2017.2-9.el8.noarch.rpm 130 kB/s | 54 kB 00:00 2023-07-14T17:32:28.549 INFO:teuthology.orchestra.run.smithi088.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 864 kB/s | 259 kB 00:00 2023-07-14T17:32:28.558 INFO:teuthology.orchestra.run.smithi178.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 173 kB/s | 34 kB 00:00 2023-07-14T17:32:28.697 INFO:teuthology.orchestra.run.smithi178.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 443 kB/s | 61 kB 00:00 2023-07-14T17:32:28.708 INFO:teuthology.orchestra.run.smithi088.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.7 MB/s | 323 kB 00:00 2023-07-14T17:32:28.742 INFO:teuthology.orchestra.run.smithi088.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 202 kB/s | 39 kB 00:00 2023-07-14T17:32:28.821 INFO:teuthology.orchestra.run.smithi178.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 622 kB/s | 196 kB 00:00 2023-07-14T17:32:28.864 INFO:teuthology.orchestra.run.smithi178.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 112 kB/s | 19 kB 00:00 2023-07-14T17:32:28.957 INFO:teuthology.orchestra.run.smithi178.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 177 kB/s | 24 kB 00:00 2023-07-14T17:32:28.984 INFO:teuthology.orchestra.run.smithi088.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 2.4 MB/s | 688 kB 00:00 2023-07-14T17:32:29.024 INFO:teuthology.orchestra.run.smithi178.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 184 kB/s | 29 kB 00:00 2023-07-14T17:32:29.050 INFO:teuthology.orchestra.run.smithi088.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 1.7 MB/s | 538 kB 00:00 2023-07-14T17:32:29.132 INFO:teuthology.orchestra.run.smithi178.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 157 kB/s | 27 kB 00:00 2023-07-14T17:32:29.216 INFO:teuthology.orchestra.run.smithi178.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 1.3 MB/s | 255 kB 00:00 2023-07-14T17:32:29.217 INFO:teuthology.orchestra.run.smithi088.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 672 kB/s | 157 kB 00:00 2023-07-14T17:32:29.347 INFO:teuthology.orchestra.run.smithi178.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.1 MB/s | 251 kB 00:00 2023-07-14T17:32:29.450 INFO:teuthology.orchestra.run.smithi178.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 262 kB/s | 61 kB 00:00 2023-07-14T17:32:29.517 INFO:teuthology.orchestra.run.smithi178.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 503 kB/s | 85 kB 00:00 2023-07-14T17:32:29.634 INFO:teuthology.orchestra.run.smithi088.stdout:(80/131): ceph-test-18.0.0-4940.g81c9387d.el8.x 4.4 MB/s | 48 MB 00:10 2023-07-14T17:32:29.642 INFO:teuthology.orchestra.run.smithi178.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 119 kB/s | 23 kB 00:00 2023-07-14T17:32:29.659 INFO:teuthology.orchestra.run.smithi178.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 4.1 MB/s | 70 kB 00:00 2023-07-14T17:32:29.660 INFO:teuthology.orchestra.run.smithi088.stdout:(81/131): oniguruma-6.8.2-2.el8.x86_64.rpm 308 kB/s | 187 kB 00:00 2023-07-14T17:32:29.702 INFO:teuthology.orchestra.run.smithi088.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 1.4 MB/s | 713 kB 00:00 2023-07-14T17:32:29.918 INFO:teuthology.orchestra.run.smithi088.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 105 kB/s | 27 kB 00:00 2023-07-14T17:32:29.944 INFO:teuthology.orchestra.run.smithi088.stdout:(84/131): flex-2.6.1-9.el8.x86_64.rpm 1.3 MB/s | 320 kB 00:00 2023-07-14T17:32:29.994 INFO:teuthology.orchestra.run.smithi088.stdout:(85/131): python3-prettytable-0.7.2-14.el8.noar 584 kB/s | 44 kB 00:00 2023-07-14T17:32:30.178 INFO:teuthology.orchestra.run.smithi088.stdout:(86/131): python3-babel-2.5.1-7.el8.noarch.rpm 8.8 MB/s | 4.8 MB 00:00 2023-07-14T17:32:30.210 INFO:teuthology.orchestra.run.smithi178.stdout:(70/131): python3-scipy-1.0.0-21.module+el8.5.0 25 MB/s | 14 MB 00:00 2023-07-14T17:32:30.245 INFO:teuthology.orchestra.run.smithi088.stdout:(87/131): python3-werkzeug-0.12.2-4.el8.noarch. 1.8 MB/s | 457 kB 00:00 2023-07-14T17:32:30.345 INFO:teuthology.orchestra.run.smithi088.stdout:(88/131): python3-numpy-1.14.3-10.el8.x86_64.rp 9.3 MB/s | 3.7 MB 00:00 2023-07-14T17:32:30.403 INFO:teuthology.orchestra.run.smithi178.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 571 kB/s | 110 kB 00:00 2023-07-14T17:32:30.437 INFO:teuthology.orchestra.run.smithi088.stdout:(89/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 870 kB/s | 225 kB 00:00 2023-07-14T17:32:30.462 INFO:teuthology.orchestra.run.smithi088.stdout:(90/131): uuid-1.6.2-43.el8.x86_64.rpm 295 kB/s | 64 kB 00:00 2023-07-14T17:32:30.544 INFO:teuthology.orchestra.run.smithi178.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 386 kB/s | 54 kB 00:00 2023-07-14T17:32:30.595 INFO:teuthology.orchestra.run.smithi178.stdout:(73/131): thrift-0.13.0-2.el8.x86_64.rpm 1.6 MB/s | 1.7 MB 00:01 2023-07-14T17:32:30.679 INFO:teuthology.orchestra.run.smithi088.stdout:(91/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 379 kB/s | 126 kB 00:00 2023-07-14T17:32:30.685 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T17:32:30.686 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T17:32:30.700 INFO:teuthology.orchestra.run.smithi107.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T17:32:30.701 INFO:teuthology.orchestra.run.smithi107.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T17:32:30.702 INFO:teuthology.orchestra.run.smithi107.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T17:32:30.704 INFO:teuthology.orchestra.run.smithi107.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T17:32:30.706 INFO:teuthology.orchestra.run.smithi107.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T17:32:30.709 INFO:teuthology.orchestra.run.smithi107.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T17:32:30.710 INFO:teuthology.orchestra.run.smithi107.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T17:32:30.779 INFO:teuthology.orchestra.run.smithi088.stdout:(92/131): python3-jsonpointer-1.10-11.el8.noarc 58 kB/s | 20 kB 00:00 2023-07-14T17:32:30.794 INFO:teuthology.orchestra.run.smithi178.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.0 MB/s | 259 kB 00:00 2023-07-14T17:32:30.804 INFO:teuthology.orchestra.run.smithi088.stdout:(93/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 302 kB/s | 103 kB 00:00 2023-07-14T17:32:30.843 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2023-07-14T17:32:30.855 INFO:teuthology.orchestra.run.smithi107.stdout:=========================================================================================================================== 2023-07-14T17:32:30.855 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2023-07-14T17:32:30.855 INFO:teuthology.orchestra.run.smithi107.stdout:=========================================================================================================================== 2023-07-14T17:32:30.855 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2023-07-14T17:32:30.855 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T17:32:30.856 INFO:teuthology.orchestra.run.smithi107.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T17:32:30.856 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T17:32:30.856 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T17:32:30.856 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T17:32:30.856 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T17:32:30.856 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T17:32:30.856 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T17:32:30.857 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T17:32:30.857 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T17:32:30.857 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T17:32:30.857 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T17:32:30.857 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T17:32:30.857 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T17:32:30.857 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T17:32:30.857 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T17:32:30.858 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T17:32:30.858 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T17:32:30.858 INFO:teuthology.orchestra.run.smithi107.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T17:32:30.858 INFO:teuthology.orchestra.run.smithi107.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T17:32:30.858 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T17:32:30.858 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T17:32:30.858 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T17:32:30.859 INFO:teuthology.orchestra.run.smithi107.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T17:32:30.859 INFO:teuthology.orchestra.run.smithi107.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T17:32:30.859 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T17:32:30.859 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T17:32:30.859 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T17:32:30.859 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T17:32:30.859 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T17:32:30.859 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T17:32:30.860 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T17:32:30.860 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T17:32:30.860 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T17:32:30.860 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T17:32:30.860 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T17:32:30.860 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T17:32:30.860 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2023-07-14T17:32:30.861 INFO:teuthology.orchestra.run.smithi107.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T17:32:30.861 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T17:32:30.861 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T17:32:30.861 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2023-07-14T17:32:30.861 INFO:teuthology.orchestra.run.smithi107.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T17:32:30.861 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T17:32:30.861 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T17:32:30.861 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T17:32:30.862 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T17:32:30.862 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T17:32:30.862 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T17:32:30.862 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T17:32:30.862 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T17:32:30.862 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T17:32:30.862 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T17:32:30.863 INFO:teuthology.orchestra.run.smithi107.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T17:32:30.863 INFO:teuthology.orchestra.run.smithi107.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T17:32:30.863 INFO:teuthology.orchestra.run.smithi107.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T17:32:30.863 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T17:32:30.863 INFO:teuthology.orchestra.run.smithi107.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T17:32:30.863 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T17:32:30.863 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T17:32:30.863 INFO:teuthology.orchestra.run.smithi107.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T17:32:30.864 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T17:32:30.864 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T17:32:30.864 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T17:32:30.864 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T17:32:30.864 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T17:32:30.864 INFO:teuthology.orchestra.run.smithi107.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T17:32:30.864 INFO:teuthology.orchestra.run.smithi107.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T17:32:30.865 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T17:32:30.865 INFO:teuthology.orchestra.run.smithi107.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T17:32:30.865 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T17:32:30.865 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T17:32:30.865 INFO:teuthology.orchestra.run.smithi107.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T17:32:30.865 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T17:32:30.865 INFO:teuthology.orchestra.run.smithi107.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T17:32:30.865 INFO:teuthology.orchestra.run.smithi107.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T17:32:30.866 INFO:teuthology.orchestra.run.smithi107.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T17:32:30.866 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T17:32:30.866 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T17:32:30.866 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T17:32:30.866 INFO:teuthology.orchestra.run.smithi107.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T17:32:30.866 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T17:32:30.866 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T17:32:30.866 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T17:32:30.867 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T17:32:30.867 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T17:32:30.867 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T17:32:30.867 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T17:32:30.867 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T17:32:30.867 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T17:32:30.867 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T17:32:30.867 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T17:32:30.868 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T17:32:30.868 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T17:32:30.868 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T17:32:30.868 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T17:32:30.868 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T17:32:30.868 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T17:32:30.868 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T17:32:30.869 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T17:32:30.869 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T17:32:30.869 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T17:32:30.869 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T17:32:30.869 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T17:32:30.869 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T17:32:30.869 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T17:32:30.869 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T17:32:30.870 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T17:32:30.870 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T17:32:30.870 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T17:32:30.870 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T17:32:30.870 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T17:32:30.870 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T17:32:30.870 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T17:32:30.871 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T17:32:30.871 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T17:32:30.871 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T17:32:30.871 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T17:32:30.871 INFO:teuthology.orchestra.run.smithi107.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T17:32:30.871 INFO:teuthology.orchestra.run.smithi107.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T17:32:30.871 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T17:32:30.871 INFO:teuthology.orchestra.run.smithi107.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T17:32:30.872 INFO:teuthology.orchestra.run.smithi107.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T17:32:30.872 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T17:32:30.872 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T17:32:30.872 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T17:32:30.872 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T17:32:30.872 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T17:32:30.872 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T17:32:30.872 INFO:teuthology.orchestra.run.smithi107.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T17:32:30.873 INFO:teuthology.orchestra.run.smithi107.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T17:32:30.873 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T17:32:30.873 INFO:teuthology.orchestra.run.smithi107.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T17:32:30.873 INFO:teuthology.orchestra.run.smithi107.stdout:Enabling module streams: 2023-07-14T17:32:30.873 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql 10 2023-07-14T17:32:30.873 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-07-14T17:32:30.873 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2023-07-14T17:32:30.874 INFO:teuthology.orchestra.run.smithi107.stdout:=========================================================================================================================== 2023-07-14T17:32:30.874 INFO:teuthology.orchestra.run.smithi107.stdout:Install 128 Packages 2023-07-14T17:32:30.874 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 3 Packages 2023-07-14T17:32:30.874 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-07-14T17:32:30.874 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 206 M 2023-07-14T17:32:30.875 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2023-07-14T17:32:31.071 INFO:teuthology.orchestra.run.smithi088.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 1.8 MB/s | 709 kB 00:00 2023-07-14T17:32:31.105 INFO:teuthology.orchestra.run.smithi088.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 659 kB/s | 198 kB 00:00 2023-07-14T17:32:31.222 INFO:teuthology.orchestra.run.smithi088.stdout:(96/131): openblas-threads-0.3.15-3.el8.x86_64. 11 MB/s | 4.9 MB 00:00 2023-07-14T17:32:31.272 INFO:teuthology.orchestra.run.smithi088.stdout:(97/131): jq-1.6-3.el8.x86_64.rpm 1.2 MB/s | 202 kB 00:00 2023-07-14T17:32:31.302 INFO:teuthology.orchestra.run.smithi178.stdout:(75/131): ceph-test-18.0.0-4940.g81c9387d.el8.x 4.8 MB/s | 48 MB 00:09 2023-07-14T17:32:31.327 INFO:teuthology.orchestra.run.smithi178.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 73 kB/s | 39 kB 00:00 2023-07-14T17:32:31.370 INFO:teuthology.orchestra.run.smithi178.stdout:(77/131): socat-1.7.4.1-1.el8.x86_64.rpm 416 kB/s | 323 kB 00:00 2023-07-14T17:32:31.431 INFO:teuthology.orchestra.run.smithi088.stdout:(98/131): openblas-0.3.15-3.el8.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2023-07-14T17:32:31.528 INFO:teuthology.orchestra.run.smithi178.stdout:(78/131): python3-mako-1.0.6-13.el8.noarch.rpm 991 kB/s | 157 kB 00:00 2023-07-14T17:32:31.556 INFO:teuthology.orchestra.run.smithi088.stdout:(99/131): librdkafka-0.11.4-3.el8.x86_64.rpm 1.0 MB/s | 354 kB 00:00 2023-07-14T17:32:31.562 INFO:teuthology.orchestra.run.smithi178.stdout:(79/131): bison-3.0.4-10.el8.x86_64.rpm 2.6 MB/s | 688 kB 00:00 2023-07-14T17:32:31.604 INFO:teuthology.orchestra.run.smithi178.stdout:(80/131): python3-jinja2-2.10.1-3.el8.noarch.rp 1.9 MB/s | 538 kB 00:00 2023-07-14T17:32:31.671 INFO:teuthology.orchestra.run.smithi178.stdout:(81/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2023-07-14T17:32:31.704 INFO:teuthology.orchestra.run.smithi178.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 4.9 MB/s | 713 kB 00:00 2023-07-14T17:32:31.781 INFO:teuthology.orchestra.run.smithi088.stdout:(100/131): postgresql-server-10.21-2.module+el8 10 MB/s | 5.1 MB 00:00 2023-07-14T17:32:31.788 INFO:teuthology.orchestra.run.smithi178.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 227 kB/s | 27 kB 00:00 2023-07-14T17:32:31.823 INFO:teuthology.orchestra.run.smithi088.stdout:(101/131): postgresql-contrib-10.21-2.module+el 2.0 MB/s | 808 kB 00:00 2023-07-14T17:32:31.847 INFO:teuthology.orchestra.run.smithi178.stdout:(84/131): flex-2.6.1-9.el8.x86_64.rpm 2.2 MB/s | 320 kB 00:00 2023-07-14T17:32:31.859 INFO:teuthology.orchestra.run.smithi107.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 14 kB/s | 6.4 kB 00:00 2023-07-14T17:32:31.972 INFO:teuthology.orchestra.run.smithi178.stdout:(85/131): python3-babel-2.5.1-7.el8.noarch.rpm 13 MB/s | 4.8 MB 00:00 2023-07-14T17:32:32.021 INFO:teuthology.orchestra.run.smithi178.stdout:(86/131): python3-prettytable-0.7.2-14.el8.noar 189 kB/s | 44 kB 00:00 2023-07-14T17:32:32.114 INFO:teuthology.orchestra.run.smithi178.stdout:(87/131): python3-werkzeug-0.12.2-4.el8.noarch. 3.2 MB/s | 457 kB 00:00 2023-07-14T17:32:32.165 INFO:teuthology.orchestra.run.smithi088.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 2.5 MB/s | 1.5 MB 00:00 2023-07-14T17:32:32.181 INFO:teuthology.orchestra.run.smithi178.stdout:(88/131): python3-numpy-1.14.3-10.el8.x86_64.rp 11 MB/s | 3.7 MB 00:00 2023-07-14T17:32:32.206 INFO:teuthology.orchestra.run.smithi178.stdout:(89/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.2 MB/s | 225 kB 00:00 2023-07-14T17:32:32.215 INFO:teuthology.orchestra.run.smithi088.stdout:(103/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 108 kB/s | 47 kB 00:00 2023-07-14T17:32:32.231 INFO:teuthology.orchestra.run.smithi178.stdout:(90/131): uuid-1.6.2-43.el8.x86_64.rpm 546 kB/s | 64 kB 00:00 2023-07-14T17:32:32.241 INFO:teuthology.orchestra.run.smithi088.stdout:(104/131): libselinux-devel-2.9-5.el8.x86_64.rp 479 kB/s | 200 kB 00:00 2023-07-14T17:32:32.365 INFO:teuthology.orchestra.run.smithi178.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 126 kB/s | 20 kB 00:00 2023-07-14T17:32:32.391 INFO:teuthology.orchestra.run.smithi178.stdout:(92/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 604 kB/s | 126 kB 00:00 2023-07-14T17:32:32.450 INFO:teuthology.orchestra.run.smithi088.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 943 kB/s | 220 kB 00:00 2023-07-14T17:32:32.475 INFO:teuthology.orchestra.run.smithi088.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 206 kB/s | 48 kB 00:00 2023-07-14T17:32:32.507 INFO:teuthology.orchestra.run.smithi178.stdout:(93/131): libtool-2.4.6-25.el8.x86_64.rpm 4.9 MB/s | 709 kB 00:00 2023-07-14T17:32:32.566 INFO:teuthology.orchestra.run.smithi178.stdout:(94/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 310 kB/s | 103 kB 00:00 2023-07-14T17:32:32.591 INFO:teuthology.orchestra.run.smithi088.stdout:(107/131): python3-jwt-1.6.1-2.el8.noarch.rpm 100 kB/s | 43 kB 00:00 2023-07-14T17:32:32.633 INFO:teuthology.orchestra.run.smithi178.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 1.5 MB/s | 198 kB 00:00 2023-07-14T17:32:32.634 INFO:teuthology.orchestra.run.smithi088.stdout:(108/131): libsepol-devel-2.9-3.el8.x86_64.rpm 473 kB/s | 87 kB 00:00 2023-07-14T17:32:32.700 INFO:teuthology.orchestra.run.smithi088.stdout:(109/131): python3-requests-oauthlib-1.0.0-1.el 189 kB/s | 43 kB 00:00 2023-07-14T17:32:32.758 INFO:teuthology.orchestra.run.smithi178.stdout:(96/131): openblas-threads-0.3.15-3.el8.x86_64. 13 MB/s | 4.9 MB 00:00 2023-07-14T17:32:32.768 INFO:teuthology.orchestra.run.smithi107.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 936 kB/s | 850 kB 00:00 2023-07-14T17:32:32.834 INFO:teuthology.orchestra.run.smithi178.stdout:(97/131): jq-1.6-3.el8.x86_64.rpm 1.0 MB/s | 202 kB 00:00 2023-07-14T17:32:32.900 INFO:teuthology.orchestra.run.smithi178.stdout:(98/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.4 MB/s | 354 kB 00:00 2023-07-14T17:32:32.925 INFO:teuthology.orchestra.run.smithi088.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 784 kB/s | 229 kB 00:00 2023-07-14T17:32:33.009 INFO:teuthology.orchestra.run.smithi088.stdout:(111/131): python3-pycparser-2.14-14.el8.noarch 261 kB/s | 109 kB 00:00 2023-07-14T17:32:33.074 INFO:teuthology.orchestra.run.smithi178.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 9.3 MB/s | 4.7 MB 00:00 2023-07-14T17:32:33.109 INFO:teuthology.orchestra.run.smithi088.stdout:(112/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 1.3 MB/s | 560 kB 00:00 2023-07-14T17:32:33.126 INFO:teuthology.orchestra.run.smithi178.stdout:(100/131): postgresql-contrib-10.21-2.module+el 3.5 MB/s | 808 kB 00:00 2023-07-14T17:32:33.201 INFO:teuthology.orchestra.run.smithi088.stdout:(113/131): libaio-devel-0.3.112-1.el8.x86_64.rp 68 kB/s | 19 kB 00:00 2023-07-14T17:32:33.235 INFO:teuthology.orchestra.run.smithi178.stdout:(101/131): postgresql-server-10.21-2.module+el8 13 MB/s | 5.1 MB 00:00 2023-07-14T17:32:33.251 INFO:teuthology.orchestra.run.smithi178.stdout:(102/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 374 kB/s | 47 kB 00:00 2023-07-14T17:32:33.277 INFO:teuthology.orchestra.run.smithi088.stdout:(114/131): python3-oauthlib-2.1.0-1.el8.noarch. 579 kB/s | 155 kB 00:00 2023-07-14T17:32:33.310 INFO:teuthology.orchestra.run.smithi088.stdout:(115/131): python3-cryptography-3.2.1-5.el8.x86 2.7 MB/s | 559 kB 00:00 2023-07-14T17:32:33.377 INFO:teuthology.orchestra.run.smithi178.stdout:(103/131): postgresql-10.21-2.module+el8.6.0+15 5.0 MB/s | 1.5 MB 00:00 2023-07-14T17:32:33.401 INFO:teuthology.orchestra.run.smithi107.stdout:(3/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 2.7 MB/s | 5.4 MB 00:02 2023-07-14T17:32:33.594 INFO:teuthology.orchestra.run.smithi088.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 46 kB/s | 18 kB 00:00 2023-07-14T17:32:33.924 INFO:teuthology.orchestra.run.smithi107.stdout:(4/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 4.4 MB/s | 2.3 MB 00:00 2023-07-14T17:32:33.924 INFO:teuthology.orchestra.run.smithi088.stdout:(117/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 11 MB/s | 2.3 MB 00:00 2023-07-14T17:32:33.925 INFO:teuthology.orchestra.run.smithi088.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 419 kB/s | 238 kB 00:00 2023-07-14T17:32:33.925 INFO:teuthology.orchestra.run.smithi088.stdout:(119/131): pcre2-devel-10.32-2.el8.x86_64.rpm 954 kB/s | 605 kB 00:00 2023-07-14T17:32:33.925 INFO:teuthology.orchestra.run.smithi178.stdout:(104/131): libselinux-devel-2.9-5.el8.x86_64.rp 479 kB/s | 200 kB 00:00 2023-07-14T17:32:33.926 INFO:teuthology.orchestra.run.smithi178.stdout:(105/131): python3-jwt-1.6.1-2.el8.noarch.rpm 93 kB/s | 43 kB 00:00 2023-07-14T17:32:33.926 INFO:teuthology.orchestra.run.smithi178.stdout:(106/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 614 kB/s | 220 kB 00:00 2023-07-14T17:32:33.926 INFO:teuthology.orchestra.run.smithi178.stdout:(107/131): keyutils-libs-devel-1.5.10-9.el8.x86 411 kB/s | 48 kB 00:00 2023-07-14T17:32:33.926 INFO:teuthology.orchestra.run.smithi178.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 300 kB/s | 43 kB 00:00 2023-07-14T17:32:33.987 INFO:teuthology.orchestra.run.smithi178.stdout:(109/131): libsepol-devel-2.9-3.el8.x86_64.rpm 315 kB/s | 87 kB 00:00 2023-07-14T17:32:33.994 INFO:teuthology.orchestra.run.smithi107.stdout:(5/131): ceph-immutable-object-cache-18.0.0-494 136 kB/s | 167 kB 00:01 2023-07-14T17:32:34.012 INFO:teuthology.orchestra.run.smithi178.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.7 MB/s | 229 kB 00:00 2023-07-14T17:32:34.037 INFO:teuthology.orchestra.run.smithi178.stdout:(111/131): python3-pycparser-2.14-14.el8.noarch 408 kB/s | 109 kB 00:00 2023-07-14T17:32:34.070 INFO:teuthology.orchestra.run.smithi088.stdout:(120/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 279 kB/s | 72 kB 00:00 2023-07-14T17:32:34.147 INFO:teuthology.orchestra.run.smithi178.stdout:(112/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.4 MB/s | 560 kB 00:00 2023-07-14T17:32:34.154 INFO:teuthology.orchestra.run.smithi088.stdout:(121/131): fuse-2.9.7-15.el8.x86_64.rpm 300 kB/s | 83 kB 00:00 2023-07-14T17:32:34.196 INFO:teuthology.orchestra.run.smithi088.stdout:(122/131): elfutils-libelf-devel-0.186-1.el8.x8 211 kB/s | 60 kB 00:00 2023-07-14T17:32:34.272 INFO:teuthology.orchestra.run.smithi178.stdout:(113/131): python3-cryptography-3.2.1-5.el8.x86 4.4 MB/s | 559 kB 00:00 2023-07-14T17:32:34.294 INFO:teuthology.orchestra.run.smithi107.stdout:(6/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 4.1 MB/s | 1.6 MB 00:00 2023-07-14T17:32:34.297 INFO:teuthology.orchestra.run.smithi178.stdout:(114/131): libaio-devel-0.3.112-1.el8.x86_64.rp 66 kB/s | 19 kB 00:00 2023-07-14T17:32:34.322 INFO:teuthology.orchestra.run.smithi178.stdout:(115/131): python3-oauthlib-2.1.0-1.el8.noarch. 545 kB/s | 155 kB 00:00 2023-07-14T17:32:34.396 INFO:teuthology.orchestra.run.smithi088.stdout:(123/131): libuuid-devel-2.32.1-35.el8.x86_64.r 301 kB/s | 98 kB 00:00 2023-07-14T17:32:34.406 INFO:teuthology.orchestra.run.smithi178.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 134 kB/s | 18 kB 00:00 2023-07-14T17:32:34.471 INFO:teuthology.orchestra.run.smithi088.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 140 kB/s | 39 kB 00:00 2023-07-14T17:32:34.481 INFO:teuthology.orchestra.run.smithi178.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.2 MB/s | 605 kB 00:00 2023-07-14T17:32:34.488 INFO:teuthology.orchestra.run.smithi088.stdout:(125/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 173 kB/s | 58 kB 00:00 2023-07-14T17:32:34.506 INFO:teuthology.orchestra.run.smithi178.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.3 MB/s | 238 kB 00:00 2023-07-14T17:32:34.582 INFO:teuthology.orchestra.run.smithi178.stdout:(119/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 13 MB/s | 2.3 MB 00:00 2023-07-14T17:32:34.616 INFO:teuthology.orchestra.run.smithi178.stdout:(120/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 540 kB/s | 72 kB 00:00 2023-07-14T17:32:34.622 INFO:teuthology.orchestra.run.smithi088.stdout:(126/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 2.8 MB/s | 644 kB 00:00 2023-07-14T17:32:34.640 INFO:teuthology.orchestra.run.smithi178.stdout:(121/131): fuse-2.9.7-15.el8.x86_64.rpm 619 kB/s | 83 kB 00:00 2023-07-14T17:32:34.664 INFO:teuthology.orchestra.run.smithi088.stdout:(127/131): NetworkManager-initscripts-updown-1. 786 kB/s | 138 kB 00:00 2023-07-14T17:32:34.666 INFO:teuthology.orchestra.run.smithi178.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-07-14T17:32:34.697 INFO:teuthology.orchestra.run.smithi088.stdout:(128/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 758 kB/s | 171 kB 00:00 2023-07-14T17:32:34.750 INFO:teuthology.orchestra.run.smithi178.stdout:(123/131): elfutils-libelf-devel-0.186-1.el8.x8 357 kB/s | 60 kB 00:00 2023-07-14T17:32:34.775 INFO:teuthology.orchestra.run.smithi178.stdout:(124/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 430 kB/s | 58 kB 00:00 2023-07-14T17:32:34.809 INFO:teuthology.orchestra.run.smithi178.stdout:(125/131): libcom_err-devel-1.45.6-4.el8.x86_64 271 kB/s | 39 kB 00:00 2023-07-14T17:32:34.867 INFO:teuthology.orchestra.run.smithi178.stdout:(126/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 5.4 MB/s | 644 kB 00:00 2023-07-14T17:32:34.892 INFO:teuthology.orchestra.run.smithi178.stdout:(127/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.4 MB/s | 171 kB 00:00 2023-07-14T17:32:34.931 INFO:teuthology.orchestra.run.smithi088.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 3.3 MB/s | 798 kB 00:00 2023-07-14T17:32:34.976 INFO:teuthology.orchestra.run.smithi178.stdout:(128/131): NetworkManager-initscripts-updown-1. 824 kB/s | 138 kB 00:00 2023-07-14T17:32:35.118 INFO:teuthology.orchestra.run.smithi178.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 5.5 MB/s | 798 kB 00:00 2023-07-14T17:32:36.281 INFO:teuthology.orchestra.run.smithi088.stdout:(130/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 2.2 MB/s | 3.6 MB 00:01 2023-07-14T17:32:36.485 INFO:teuthology.orchestra.run.smithi107.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 1.9 MB/s | 4.6 MB 00:02 2023-07-14T17:32:36.864 INFO:teuthology.orchestra.run.smithi088.stdout:(131/131): librados2-18.0.0-4940.g81c9387d.el8. 1.6 MB/s | 3.7 MB 00:02 2023-07-14T17:32:36.867 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-07-14T17:32:36.867 INFO:teuthology.orchestra.run.smithi088.stdout:Total 9.2 MB/s | 206 MB 00:22 2023-07-14T17:32:36.985 INFO:teuthology.orchestra.run.smithi178.stdout:(130/131): librados2-18.0.0-4940.g81c9387d.el8. 1.7 MB/s | 3.7 MB 00:02 2023-07-14T17:32:37.451 INFO:teuthology.orchestra.run.smithi178.stdout:(131/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 1.4 MB/s | 3.6 MB 00:02 2023-07-14T17:32:37.454 INFO:teuthology.orchestra.run.smithi178.stdout:-------------------------------------------------------------------------------- 2023-07-14T17:32:37.454 INFO:teuthology.orchestra.run.smithi178.stdout:Total 9.1 MB/s | 206 MB 00:22 2023-07-14T17:32:37.670 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-07-14T17:32:37.790 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-07-14T17:32:37.791 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-07-14T17:32:37.928 INFO:teuthology.orchestra.run.smithi107.stdout:(8/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x86 4.9 MB/s | 18 MB 00:03 2023-07-14T17:32:38.069 INFO:teuthology.orchestra.run.smithi107.stdout:(9/131): ceph-selinux-18.0.0-4940.g81c9387d.el8 172 kB/s | 24 kB 00:00 2023-07-14T17:32:38.296 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction check 2023-07-14T17:32:38.416 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction check succeeded. 2023-07-14T17:32:38.417 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction test 2023-07-14T17:32:38.898 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-07-14T17:32:38.902 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-07-14T17:32:39.523 INFO:teuthology.orchestra.run.smithi178.stdout:Transaction test succeeded. 2023-07-14T17:32:39.527 INFO:teuthology.orchestra.run.smithi178.stdout:Running transaction 2023-07-14T17:32:42.510 INFO:teuthology.orchestra.run.smithi107.stdout:(10/131): ceph-radosgw-18.0.0-4940.g81c9387d.el 2.2 MB/s | 14 MB 00:06 2023-07-14T17:32:42.960 INFO:teuthology.orchestra.run.smithi107.stdout:(11/131): ceph-common-18.0.0-4940.g81c9387d.el8 1.9 MB/s | 22 MB 00:11 2023-07-14T17:32:42.987 INFO:teuthology.orchestra.run.smithi107.stdout:(12/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 582 kB/s | 260 kB 00:00 2023-07-14T17:32:43.103 INFO:teuthology.orchestra.run.smithi107.stdout:(13/131): libcephfs-devel-18.0.0-4940.g81c9387d 219 kB/s | 31 kB 00:00 2023-07-14T17:32:43.236 INFO:teuthology.orchestra.run.smithi107.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 1.0 MB/s | 135 kB 00:00 2023-07-14T17:32:43.287 INFO:teuthology.orchestra.run.smithi107.stdout:(15/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 2.5 MB/s | 772 kB 00:00 2023-07-14T17:32:43.362 INFO:teuthology.orchestra.run.smithi107.stdout:(16/131): librados-devel-18.0.0-4940.g81c9387d. 1.1 MB/s | 139 kB 00:00 2023-07-14T17:32:43.479 INFO:teuthology.orchestra.run.smithi107.stdout:(17/131): libradosstriper1-18.0.0-4940.g81c9387 2.4 MB/s | 473 kB 00:00 2023-07-14T17:32:43.579 INFO:teuthology.orchestra.run.smithi107.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 447 kB/s | 45 kB 00:00 2023-07-14T17:32:43.662 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-07-14T17:32:43.696 INFO:teuthology.orchestra.run.smithi107.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 1.0 MB/s | 122 kB 00:00 2023-07-14T17:32:43.805 INFO:teuthology.orchestra.run.smithi107.stdout:(20/131): python3-cephfs-18.0.0-4940.g81c9387d. 1.8 MB/s | 197 kB 00:00 2023-07-14T17:32:43.871 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T17:32:43.988 INFO:teuthology.orchestra.run.smithi107.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 2.0 MB/s | 371 kB 00:00 2023-07-14T17:32:44.041 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T17:32:44.214 INFO:teuthology.orchestra.run.smithi107.stdout:(22/131): python3-rbd-18.0.0-4940.g81c9387d.el8 1.6 MB/s | 364 kB 00:00 2023-07-14T17:32:44.263 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-07-14T17:32:44.339 INFO:teuthology.orchestra.run.smithi107.stdout:(23/131): python3-rgw-18.0.0-4940.g81c9387d.el8 853 kB/s | 107 kB 00:00 2023-07-14T17:32:44.448 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-14T17:32:44.456 INFO:teuthology.orchestra.run.smithi107.stdout:(24/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 802 kB/s | 94 kB 00:00 2023-07-14T17:32:44.554 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-14T17:32:44.787 INFO:teuthology.orchestra.run.smithi178.stdout: Preparing : 1/1 2023-07-14T17:32:44.814 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-14T17:32:45.003 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T17:32:45.199 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-14T17:32:45.205 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T17:32:45.206 INFO:teuthology.orchestra.run.smithi107.stdout:(25/131): ceph-test-18.0.0-4940.g81c9387d.el8.x 6.7 MB/s | 48 MB 00:07 2023-07-14T17:32:45.306 INFO:teuthology.orchestra.run.smithi107.stdout:(26/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 3.5 MB/s | 6.8 MB 00:01 2023-07-14T17:32:45.332 INFO:teuthology.orchestra.run.smithi107.stdout:(27/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 1.4 MB/s | 179 kB 00:00 2023-07-14T17:32:45.414 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T17:32:45.428 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-07-14T17:32:45.432 INFO:teuthology.orchestra.run.smithi107.stdout:(28/131): ceph-grafana-dashboards-18.0.0-4940.g 184 kB/s | 23 kB 00:00 2023-07-14T17:32:45.457 INFO:teuthology.orchestra.run.smithi107.stdout:(29/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 1.0 MB/s | 129 kB 00:00 2023-07-14T17:32:45.571 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-14T17:32:45.674 INFO:teuthology.orchestra.run.smithi107.stdout:(30/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 2.9 MB/s | 3.5 MB 00:01 2023-07-14T17:32:45.710 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-14T17:32:45.791 INFO:teuthology.orchestra.run.smithi107.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 2.0 MB/s | 241 kB 00:00 2023-07-14T17:32:45.807 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T17:32:45.883 INFO:teuthology.orchestra.run.smithi107.stdout:(32/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 3.8 MB/s | 1.7 MB 00:00 2023-07-14T17:32:45.908 INFO:teuthology.orchestra.run.smithi107.stdout:(33/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 405 kB/s | 47 kB 00:00 2023-07-14T17:32:45.988 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-14T17:32:46.000 INFO:teuthology.orchestra.run.smithi107.stdout:(34/131): ceph-prometheus-alerts-18.0.0-4940.g8 124 kB/s | 15 kB 00:00 2023-07-14T17:32:46.006 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T17:32:46.017 INFO:teuthology.orchestra.run.smithi107.stdout:(35/131): ceph-volume-18.0.0-4940.g81c9387d.el8 2.3 MB/s | 258 kB 00:00 2023-07-14T17:32:46.118 INFO:teuthology.orchestra.run.smithi107.stdout:(36/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 1.4 MB/s | 165 kB 00:00 2023-07-14T17:32:46.129 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T17:32:46.143 INFO:teuthology.orchestra.run.smithi107.stdout:(37/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 143 kB/s | 18 kB 00:00 2023-07-14T17:32:46.322 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T17:32:46.389 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-14T17:32:46.418 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T17:32:46.510 INFO:teuthology.orchestra.run.smithi107.stdout:(38/131): ceph-mgr-diskprediction-local-18.0.0- 7.0 MB/s | 7.4 MB 00:01 2023-07-14T17:32:46.568 INFO:teuthology.orchestra.run.smithi107.stdout:(39/131): liboath-2.6.2-3.el8.x86_64.rpm 138 kB/s | 59 kB 00:00 2023-07-14T17:32:46.594 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T17:32:46.604 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T17:32:46.777 INFO:teuthology.orchestra.run.smithi107.stdout:(40/131): gperftools-libs-2.7-9.el8.x86_64.rpm 464 kB/s | 306 kB 00:00 2023-07-14T17:32:46.834 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T17:32:46.902 INFO:teuthology.orchestra.run.smithi107.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 349 kB/s | 44 kB 00:00 2023-07-14T17:32:46.927 INFO:teuthology.orchestra.run.smithi107.stdout:(42/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.3 MB/s | 476 kB 00:00 2023-07-14T17:32:46.944 INFO:teuthology.orchestra.run.smithi107.stdout:(43/131): libunwind-1.3.1-3.el8.x86_64.rpm 173 kB/s | 75 kB 00:00 2023-07-14T17:32:47.020 INFO:teuthology.orchestra.run.smithi107.stdout:(44/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.5 MB/s | 185 kB 00:00 2023-07-14T17:32:47.025 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T17:32:47.044 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T17:32:47.053 INFO:teuthology.orchestra.run.smithi107.stdout:(45/131): python3-cachetools-3.1.1-4.el8.noarch 262 kB/s | 33 kB 00:00 2023-07-14T17:32:47.070 INFO:teuthology.orchestra.run.smithi107.stdout:(46/131): python3-certifi-2018.10.15-7.el8.noar 129 kB/s | 16 kB 00:00 2023-07-14T17:32:47.154 INFO:teuthology.orchestra.run.smithi107.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.3 MB/s | 173 kB 00:00 2023-07-14T17:32:47.188 INFO:teuthology.orchestra.run.smithi107.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 695 kB/s | 81 kB 00:00 2023-07-14T17:32:47.254 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T17:32:47.271 INFO:teuthology.orchestra.run.smithi107.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 95 kB/s | 11 kB 00:00 2023-07-14T17:32:47.297 INFO:teuthology.orchestra.run.smithi107.stdout:(50/131): python3-cherrypy-18.4.0-1.el8.noarch. 1.5 MB/s | 384 kB 00:00 2023-07-14T17:32:47.322 INFO:teuthology.orchestra.run.smithi107.stdout:(51/131): python3-jaraco-functools-2.0-4.el8.no 131 kB/s | 18 kB 00:00 2023-07-14T17:32:47.368 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T17:32:47.430 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T17:32:47.439 INFO:teuthology.orchestra.run.smithi107.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 343 kB/s | 49 kB 00:00 2023-07-14T17:32:47.464 INFO:teuthology.orchestra.run.smithi107.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 413 kB/s | 59 kB 00:00 2023-07-14T17:32:47.562 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T17:32:47.597 INFO:teuthology.orchestra.run.smithi107.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 380 kB/s | 60 kB 00:00 2023-07-14T17:32:47.623 INFO:teuthology.orchestra.run.smithi107.stdout:(55/131): python3-kubernetes-11.0.0-6.el8.noarc 3.3 MB/s | 1.2 MB 00:00 2023-07-14T17:32:47.648 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T17:32:47.700 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T17:32:47.706 INFO:teuthology.orchestra.run.smithi107.stdout:(56/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.1 MB/s | 283 kB 00:00 2023-07-14T17:32:47.732 INFO:teuthology.orchestra.run.smithi107.stdout:(57/131): python3-portend-2.6-1.el8.noarch.rpm 116 kB/s | 16 kB 00:00 2023-07-14T17:32:47.757 INFO:teuthology.orchestra.run.smithi107.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 251 kB/s | 34 kB 00:00 2023-07-14T17:32:47.832 INFO:teuthology.orchestra.run.smithi107.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 1.5 MB/s | 196 kB 00:00 2023-07-14T17:32:47.846 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T17:32:47.847 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T17:32:47.866 INFO:teuthology.orchestra.run.smithi107.stdout:(60/131): python3-rsa-4.8-1.el8.noarch.rpm 460 kB/s | 61 kB 00:00 2023-07-14T17:32:47.891 INFO:teuthology.orchestra.run.smithi107.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 140 kB/s | 19 kB 00:00 2023-07-14T17:32:47.952 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T17:32:47.958 INFO:teuthology.orchestra.run.smithi107.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 191 kB/s | 24 kB 00:00 2023-07-14T17:32:48.025 INFO:teuthology.orchestra.run.smithi107.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 184 kB/s | 29 kB 00:00 2023-07-14T17:32:48.042 INFO:teuthology.orchestra.run.smithi107.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 183 kB/s | 27 kB 00:00 2023-07-14T17:32:48.100 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T17:32:48.105 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T17:32:48.185 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T17:32:48.230 INFO:teuthology.orchestra.run.smithi107.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 940 kB/s | 255 kB 00:00 2023-07-14T17:32:48.332 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T17:32:48.441 INFO:teuthology.orchestra.run.smithi107.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 153 kB/s | 61 kB 00:00 2023-07-14T17:32:48.476 INFO:teuthology.orchestra.run.smithi107.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 347 kB/s | 85 kB 00:00 2023-07-14T17:32:48.501 INFO:teuthology.orchestra.run.smithi107.stdout:(68/131): python3-webob-1.8.5-1.el8.1.noarch.rp 528 kB/s | 251 kB 00:00 2023-07-14T17:32:48.527 INFO:teuthology.orchestra.run.smithi107.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-14T17:32:48.569 INFO:teuthology.orchestra.run.smithi107.stdout:(70/131): python3-zc-lockfile-2.0-2.el8.noarch. 180 kB/s | 23 kB 00:00 2023-07-14T17:32:48.638 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-14T17:32:48.652 INFO:teuthology.orchestra.run.smithi107.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 374 kB/s | 47 kB 00:00 2023-07-14T17:32:48.719 INFO:teuthology.orchestra.run.smithi107.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.3 MB/s | 200 kB 00:00 2023-07-14T17:32:48.759 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T17:32:48.770 INFO:teuthology.orchestra.run.smithi107.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 363 kB/s | 43 kB 00:00 2023-07-14T17:32:48.781 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-14T17:32:48.828 INFO:teuthology.orchestra.run.smithi107.stdout:(74/131): thrift-0.13.0-2.el8.x86_64.rpm 4.9 MB/s | 1.7 MB 00:00 2023-07-14T17:32:48.904 INFO:teuthology.orchestra.run.smithi107.stdout:(75/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.2 MB/s | 220 kB 00:00 2023-07-14T17:32:48.929 INFO:teuthology.orchestra.run.smithi107.stdout:(76/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 303 kB/s | 48 kB 00:00 2023-07-14T17:32:48.946 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-14T17:32:48.982 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T17:32:48.996 INFO:teuthology.orchestra.run.smithi107.stdout:(77/131): libsepol-devel-2.9-3.el8.x86_64.rpm 520 kB/s | 87 kB 00:00 2023-07-14T17:32:49.030 INFO:teuthology.orchestra.run.smithi107.stdout:(78/131): python3-requests-oauthlib-1.0.0-1.el8 340 kB/s | 43 kB 00:00 2023-07-14T17:32:49.105 INFO:teuthology.orchestra.run.smithi107.stdout:(79/131): python3-pycparser-2.14-14.el8.noarch. 621 kB/s | 109 kB 00:00 2023-07-14T17:32:49.137 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T17:32:49.147 INFO:teuthology.orchestra.run.smithi107.stdout:(80/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.5 MB/s | 229 kB 00:00 2023-07-14T17:32:49.222 INFO:teuthology.orchestra.run.smithi107.stdout:(81/131): python3-oauthlib-2.1.0-1.el8.noarch.r 2.0 MB/s | 155 kB 00:00 2023-07-14T17:32:49.249 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T17:32:49.256 INFO:teuthology.orchestra.run.smithi107.stdout:(82/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 2.4 MB/s | 560 kB 00:00 2023-07-14T17:32:49.323 INFO:teuthology.orchestra.run.smithi107.stdout:(83/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 86 kB/s | 19 kB 00:00 2023-07-14T17:32:49.373 INFO:teuthology.orchestra.run.smithi107.stdout:(84/131): python3-cryptography-3.2.1-5.el8.x86_ 3.6 MB/s | 559 kB 00:00 2023-07-14T17:32:49.399 INFO:teuthology.orchestra.run.smithi107.stdout:(85/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 126 kB/s | 18 kB 00:00 2023-07-14T17:32:49.404 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T17:32:49.440 INFO:teuthology.orchestra.run.smithi107.stdout:(86/131): pcre2-devel-10.32-2.el8.x86_64.rpm 5.0 MB/s | 605 kB 00:00 2023-07-14T17:32:49.486 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T17:32:49.508 INFO:teuthology.orchestra.run.smithi107.stdout:(87/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.7 MB/s | 238 kB 00:00 2023-07-14T17:32:49.562 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-14T17:32:49.645 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-14T17:32:49.683 INFO:teuthology.orchestra.run.smithi107.stdout:(88/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 8.2 MB/s | 2.3 MB 00:00 2023-07-14T17:32:49.713 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-14T17:32:49.750 INFO:teuthology.orchestra.run.smithi107.stdout:(89/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 234 kB/s | 72 kB 00:00 2023-07-14T17:32:49.783 INFO:teuthology.orchestra.run.smithi107.stdout:(90/131): fuse-2.9.7-15.el8.x86_64.rpm 300 kB/s | 83 kB 00:00 2023-07-14T17:32:49.796 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-14T17:32:49.825 INFO:teuthology.orchestra.run.smithi107.stdout:(91/131): elfutils-libelf-devel-0.186-1.el8.x86 421 kB/s | 60 kB 00:00 2023-07-14T17:32:49.876 INFO:teuthology.orchestra.run.smithi107.stdout:(92/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 782 kB/s | 98 kB 00:00 2023-07-14T17:32:49.953 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-14T17:32:49.984 INFO:teuthology.orchestra.run.smithi107.stdout:(93/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 287 kB/s | 58 kB 00:00 2023-07-14T17:32:50.009 INFO:teuthology.orchestra.run.smithi107.stdout:(94/131): libcom_err-devel-1.45.6-4.el8.x86_64. 210 kB/s | 39 kB 00:00 2023-07-14T17:32:50.060 INFO:teuthology.orchestra.run.smithi107.stdout:(95/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 3.4 MB/s | 644 kB 00:00 2023-07-14T17:32:50.085 INFO:teuthology.orchestra.run.smithi107.stdout:(96/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.7 MB/s | 171 kB 00:00 2023-07-14T17:32:50.118 INFO:teuthology.orchestra.run.smithi107.stdout:(97/131): NetworkManager-initscripts-updown-1.3 1.2 MB/s | 138 kB 00:00 2023-07-14T17:32:50.244 INFO:teuthology.orchestra.run.smithi107.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 694 kB/s | 110 kB 00:00 2023-07-14T17:32:50.286 INFO:teuthology.orchestra.run.smithi107.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 324 kB/s | 54 kB 00:00 2023-07-14T17:32:50.311 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-14T17:32:50.411 INFO:teuthology.orchestra.run.smithi107.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.5 MB/s | 259 kB 00:00 2023-07-14T17:32:50.538 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-07-14T17:32:50.552 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-14T17:32:50.606 INFO:teuthology.orchestra.run.smithi107.stdout:(101/131): python3-scipy-1.0.0-21.module+el8.5. 25 MB/s | 14 MB 00:00 2023-07-14T17:32:50.637 INFO:teuthology.orchestra.run.smithi107.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 172 kB/s | 39 kB 00:00 2023-07-14T17:32:50.662 INFO:teuthology.orchestra.run.smithi107.stdout:(103/131): socat-1.7.4.1-1.el8.x86_64.rpm 859 kB/s | 323 kB 00:00 2023-07-14T17:32:50.729 INFO:teuthology.orchestra.run.smithi107.stdout:(104/131): bison-3.0.4-10.el8.x86_64.rpm 5.5 MB/s | 688 kB 00:00 2023-07-14T17:32:50.737 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-14T17:32:50.796 INFO:teuthology.orchestra.run.smithi107.stdout:(105/131): python3-jinja2-2.10.1-3.el8.noarch.r 3.3 MB/s | 538 kB 00:00 2023-07-14T17:32:50.821 INFO:teuthology.orchestra.run.smithi107.stdout:(106/131): python3-mako-1.0.6-13.el8.noarch.rpm 987 kB/s | 157 kB 00:00 2023-07-14T17:32:50.888 INFO:teuthology.orchestra.run.smithi107.stdout:(107/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2023-07-14T17:32:50.938 INFO:teuthology.orchestra.run.smithi107.stdout:(108/131): automake-1.16.1-7.el8.noarch.rpm 4.9 MB/s | 713 kB 00:00 2023-07-14T17:32:51.072 INFO:teuthology.orchestra.run.smithi107.stdout:(109/131): python3-babel-2.5.1-7.el8.noarch.rpm 19 MB/s | 4.8 MB 00:00 2023-07-14T17:32:51.097 INFO:teuthology.orchestra.run.smithi107.stdout:(110/131): python3-jsonpatch-1.21-2.el8.noarch. 127 kB/s | 27 kB 00:00 2023-07-14T17:32:51.130 INFO:teuthology.orchestra.run.smithi107.stdout:(111/131): flex-2.6.1-9.el8.x86_64.rpm 1.6 MB/s | 320 kB 00:00 2023-07-14T17:32:51.213 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-07-14T17:32:51.248 INFO:teuthology.orchestra.run.smithi107.stdout:(112/131): python3-prettytable-0.7.2-14.el8.noa 251 kB/s | 44 kB 00:00 2023-07-14T17:32:51.281 INFO:teuthology.orchestra.run.smithi107.stdout:(113/131): python3-werkzeug-0.12.2-4.el8.noarch 3.0 MB/s | 457 kB 00:00 2023-07-14T17:32:51.361 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-14T17:32:51.364 INFO:teuthology.orchestra.run.smithi107.stdout:(114/131): python3-numpy-1.14.3-10.el8.x86_64.r 14 MB/s | 3.7 MB 00:00 2023-07-14T17:32:51.390 INFO:teuthology.orchestra.run.smithi107.stdout:(115/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.5 MB/s | 225 kB 00:00 2023-07-14T17:32:51.473 INFO:teuthology.orchestra.run.smithi107.stdout:(116/131): uuid-1.6.2-43.el8.x86_64.rpm 333 kB/s | 64 kB 00:00 2023-07-14T17:32:51.486 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-07-14T17:32:51.490 INFO:teuthology.orchestra.run.smithi107.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 199 kB/s | 20 kB 00:00 2023-07-14T17:32:51.516 INFO:teuthology.orchestra.run.smithi107.stdout:(118/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 838 kB/s | 126 kB 00:00 2023-07-14T17:32:51.536 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-07-14T17:32:51.616 INFO:teuthology.orchestra.run.smithi107.stdout:(119/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 727 kB/s | 103 kB 00:00 2023-07-14T17:32:51.650 INFO:teuthology.orchestra.run.smithi107.stdout:(120/131): libtool-2.4.6-25.el8.x86_64.rpm 4.4 MB/s | 709 kB 00:00 2023-07-14T17:32:51.704 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-07-14T17:32:51.708 INFO:teuthology.orchestra.run.smithi107.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 2.1 MB/s | 198 kB 00:00 2023-07-14T17:32:51.775 INFO:teuthology.orchestra.run.smithi107.stdout:(122/131): openblas-threads-0.3.15-3.el8.x86_64 19 MB/s | 4.9 MB 00:00 2023-07-14T17:32:51.817 INFO:teuthology.orchestra.run.smithi107.stdout:(123/131): jq-1.6-3.el8.x86_64.rpm 1.8 MB/s | 202 kB 00:00 2023-07-14T17:32:51.916 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-07-14T17:32:51.967 INFO:teuthology.orchestra.run.smithi107.stdout:(124/131): openblas-0.3.15-3.el8.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2023-07-14T17:32:52.009 INFO:teuthology.orchestra.run.smithi107.stdout:(125/131): librdkafka-0.11.4-3.el8.x86_64.rpm 1.5 MB/s | 354 kB 00:00 2023-07-14T17:32:52.109 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-07-14T17:32:52.110 INFO:teuthology.orchestra.run.smithi107.stdout:(126/131): postgresql-server-10.21-2.module+el8 17 MB/s | 5.1 MB 00:00 2023-07-14T17:32:52.152 INFO:teuthology.orchestra.run.smithi107.stdout:(127/131): postgresql-contrib-10.21-2.module+el 4.3 MB/s | 808 kB 00:00 2023-07-14T17:32:52.193 INFO:teuthology.orchestra.run.smithi107.stdout:(128/131): postgresql-10.21-2.module+el8.6.0+15 8.2 MB/s | 1.5 MB 00:00 2023-07-14T17:32:52.281 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-07-14T17:32:52.317 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-07-14T17:32:52.394 INFO:teuthology.orchestra.run.smithi107.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 3.9 MB/s | 798 kB 00:00 2023-07-14T17:32:52.559 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-07-14T17:32:52.580 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T17:32:52.760 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T17:32:52.813 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-07-14T17:32:52.898 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T17:32:52.969 INFO:teuthology.orchestra.run.smithi107.stdout:(130/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 4.4 MB/s | 3.6 MB 00:00 2023-07-14T17:32:53.065 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-07-14T17:32:53.284 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-07-14T17:32:53.294 INFO:teuthology.orchestra.run.smithi107.stdout:(131/131): librados2-18.0.0-4940.g81c9387d.el8. 3.1 MB/s | 3.7 MB 00:01 2023-07-14T17:32:53.297 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2023-07-14T17:32:53.297 INFO:teuthology.orchestra.run.smithi107.stdout:Total 9.2 MB/s | 206 MB 00:22 2023-07-14T17:32:53.542 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-07-14T17:32:53.796 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T17:32:53.820 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T17:32:53.984 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T17:32:54.097 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T17:32:54.133 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-07-14T17:32:54.146 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2023-07-14T17:32:54.266 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2023-07-14T17:32:54.266 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2023-07-14T17:32:54.405 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-07-14T17:32:54.647 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-07-14T17:32:54.837 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-14T17:32:55.034 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T17:32:55.333 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-07-14T17:32:55.389 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2023-07-14T17:32:55.393 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2023-07-14T17:32:55.646 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-07-14T17:32:55.926 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-07-14T17:32:56.145 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-14T17:32:56.609 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-14T17:32:56.937 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-07-14T17:32:57.155 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T17:32:57.379 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T17:32:57.567 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T17:32:57.731 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T17:32:57.894 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-14T17:32:57.901 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-07-14T17:32:58.063 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-07-14T17:32:58.221 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-07-14T17:32:58.297 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-07-14T17:32:58.456 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-07-14T17:32:58.481 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T17:32:58.708 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-07-14T17:32:58.712 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T17:32:58.915 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-07-14T17:32:58.949 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T17:32:59.123 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T17:32:59.309 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-07-14T17:32:59.329 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-07-14T17:32:59.546 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-07-14T17:32:59.622 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-07-14T17:32:59.763 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-07-14T17:32:59.829 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-07-14T17:32:59.897 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-07-14T17:33:00.064 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-07-14T17:33:00.290 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-07-14T17:33:00.509 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-07-14T17:33:00.609 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-07-14T17:33:00.627 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2023-07-14T17:33:00.742 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-07-14T17:33:00.803 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T17:33:00.843 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-07-14T17:33:01.044 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T17:33:01.069 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-07-14T17:33:01.143 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-07-14T17:33:01.205 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T17:33:01.290 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-07-14T17:33:01.318 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T17:33:01.473 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-07-14T17:33:01.480 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-07-14T17:33:01.597 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T17:33:01.639 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-07-14T17:33:01.678 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-07-14T17:33:01.925 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-07-14T17:33:01.967 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-07-14T17:33:01.994 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T17:33:02.213 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T17:33:02.223 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-07-14T17:33:02.233 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-07-14T17:33:02.489 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-07-14T17:33:02.583 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-07-14T17:33:02.595 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T17:33:02.742 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-07-14T17:33:02.805 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T17:33:02.817 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-07-14T17:33:02.917 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T17:33:02.942 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-07-14T17:33:02.985 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-07-14T17:33:03.088 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T17:33:03.098 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-07-14T17:33:03.181 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T17:33:03.221 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-07-14T17:33:03.268 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-07-14T17:33:03.334 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T17:33:03.338 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-07-14T17:33:03.404 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-07-14T17:33:03.505 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-14T17:33:03.584 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-07-14T17:33:03.595 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-07-14T17:33:03.775 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T17:33:03.779 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-07-14T17:33:03.880 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-07-14T17:33:03.968 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-14T17:33:03.982 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T17:33:04.048 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-14T17:33:04.171 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-07-14T17:33:04.272 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-07-14T17:33:04.410 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T17:33:04.430 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-07-14T17:33:04.460 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-07-14T17:33:04.617 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T17:33:04.674 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-07-14T17:33:04.683 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T17:33:04.813 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T17:33:04.820 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T17:33:04.824 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-07-14T17:33:04.918 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T17:33:05.003 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-07-14T17:33:05.087 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T17:33:05.104 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T17:33:05.184 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-07-14T17:33:05.230 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T17:33:05.243 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-07-14T17:33:05.333 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T17:33:05.466 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-14T17:33:05.497 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T17:33:05.502 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-07-14T17:33:05.643 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-14T17:33:05.653 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T17:33:05.729 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-07-14T17:33:05.849 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-14T17:33:05.858 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-14T17:33:05.885 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T17:33:05.978 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T17:33:05.980 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-14T17:33:06.080 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T17:33:06.124 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T17:33:06.196 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-07-14T17:33:06.262 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T17:33:06.312 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T17:33:06.359 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-07-14T17:33:06.429 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T17:33:06.533 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T17:33:06.574 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T17:33:06.711 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T17:33:06.790 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T17:33:06.911 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T17:33:06.967 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T17:33:07.013 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T17:33:07.101 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T17:33:07.198 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T17:33:07.223 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T17:33:07.362 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T17:33:07.472 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T17:33:07.522 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T17:33:07.683 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T17:33:07.702 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T17:33:07.724 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T17:33:07.870 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T17:33:07.884 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T17:33:07.977 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T17:33:08.102 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T17:33:08.184 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T17:33:08.369 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T17:33:08.496 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T17:33:08.608 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T17:33:08.733 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T17:33:08.886 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-14T17:33:09.002 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T17:33:09.119 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-14T17:33:09.202 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T17:33:09.213 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T17:33:09.389 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T17:33:09.424 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T17:33:09.528 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T17:33:09.560 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T17:33:09.640 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T17:33:09.701 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T17:33:09.822 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T17:33:09.841 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T17:33:09.942 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T17:33:10.071 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T17:33:10.167 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T17:33:10.194 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T17:33:10.450 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T17:33:10.683 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T17:33:11.168 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T17:33:11.458 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T17:33:11.461 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T17:33:11.649 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T17:33:11.724 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T17:33:11.725 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T17:33:11.883 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T17:33:11.958 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T17:33:12.066 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T17:33:12.383 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T17:33:12.571 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T17:33:12.952 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-14T17:33:13.259 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-14T17:33:13.386 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T17:33:13.530 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-14T17:33:13.557 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T17:33:13.667 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T17:33:13.782 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-14T17:33:13.824 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T17:33:13.988 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T17:33:14.195 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T17:33:14.366 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T17:33:14.550 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T17:33:14.709 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-14T17:33:14.869 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-14T17:33:15.076 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T17:33:15.128 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T17:33:15.346 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-14T17:33:15.446 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T17:33:15.580 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-14T17:33:15.682 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T17:33:15.735 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T17:33:15.848 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T17:33:16.064 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-14T17:33:16.244 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-14T17:33:16.271 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T17:33:16.408 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-14T17:33:16.570 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-14T17:33:16.614 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T17:33:16.701 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T17:33:16.770 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-14T17:33:16.856 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T17:33:16.921 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-14T17:33:17.150 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-14T17:33:17.348 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-14T17:33:17.569 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T17:33:17.748 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-14T17:33:17.765 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T17:33:18.069 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-14T17:33:18.313 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-14T17:33:18.363 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T17:33:18.564 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-14T17:33:18.659 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T17:33:18.747 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T17:33:18.781 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-14T17:33:18.948 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T17:33:19.092 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-14T17:33:19.358 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-14T17:33:19.559 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-14T17:33:19.951 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-14T17:33:20.329 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-14T17:33:20.495 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-14T17:33:20.816 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-14T17:33:21.080 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-14T17:33:21.321 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-14T17:33:21.576 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-14T17:33:21.733 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T17:33:21.953 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T17:33:22.193 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T17:33:22.465 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T17:33:22.604 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T17:33:22.761 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T17:33:23.002 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T17:33:23.196 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T17:33:23.403 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T17:33:23.578 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T17:33:23.764 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T17:33:23.996 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T17:33:24.279 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T17:33:24.455 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T17:33:24.713 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T17:33:24.965 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T17:33:25.130 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T17:33:25.355 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T17:33:26.502 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T17:33:26.819 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T17:33:27.057 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T17:33:27.408 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T17:33:27.715 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T17:33:27.912 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T17:33:28.734 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T17:33:28.940 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T17:33:29.012 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T17:33:29.173 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T17:33:32.847 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T17:33:33.057 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T17:33:33.635 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T17:33:33.951 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T17:33:34.032 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T17:33:34.189 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T17:33:35.393 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T17:33:35.393 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /sys 2023-07-14T17:33:35.393 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /proc 2023-07-14T17:33:35.394 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /dev 2023-07-14T17:33:35.394 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /run 2023-07-14T17:33:35.394 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /mnt 2023-07-14T17:33:35.394 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /var/tmp 2023-07-14T17:33:35.394 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /home 2023-07-14T17:33:35.394 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /tmp 2023-07-14T17:33:35.394 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /dev 2023-07-14T17:33:35.394 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:33:35.769 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T17:33:37.350 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T17:33:37.534 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T17:33:37.753 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T17:33:37.788 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T17:33:37.789 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /sys 2023-07-14T17:33:37.789 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /proc 2023-07-14T17:33:37.789 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /dev 2023-07-14T17:33:37.789 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /run 2023-07-14T17:33:37.789 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /mnt 2023-07-14T17:33:37.790 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /var/tmp 2023-07-14T17:33:37.790 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /home 2023-07-14T17:33:37.790 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /tmp 2023-07-14T17:33:37.790 INFO:teuthology.orchestra.run.smithi178.stdout:skipping the directory /dev 2023-07-14T17:33:37.790 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-07-14T17:33:37.926 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T17:33:38.177 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T17:33:38.379 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T17:33:38.583 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T17:33:38.600 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T17:33:39.756 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T17:33:39.963 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T17:33:40.183 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T17:33:40.231 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T17:33:40.352 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T17:33:40.427 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T17:33:40.582 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T17:33:40.774 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T17:33:40.806 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T17:33:40.950 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T17:33:41.002 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T17:33:41.020 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T17:33:41.166 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T17:33:41.474 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T17:33:42.431 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T17:33:42.600 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T17:33:42.661 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T17:33:42.692 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T17:33:42.873 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T17:33:43.006 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T17:33:43.077 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T17:33:43.191 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T17:33:43.299 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T17:33:43.334 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T17:33:43.417 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T17:33:43.533 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T17:33:43.544 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T17:33:43.633 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T17:33:43.904 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T17:33:44.852 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T17:33:44.864 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T17:33:45.018 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T17:33:45.051 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T17:33:45.084 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T17:33:45.109 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T17:33:45.242 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T17:33:45.283 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T17:33:45.466 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T17:33:45.515 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T17:33:45.761 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T17:33:45.796 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T17:33:45.817 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T17:33:45.975 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T17:33:45.992 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T17:33:46.004 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T17:33:47.294 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T17:33:47.518 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T17:33:47.552 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T17:33:47.725 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T17:33:47.767 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T17:33:47.908 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T17:33:48.260 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T17:33:48.408 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T17:33:50.509 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T17:33:50.809 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T17:33:50.982 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-07-14T17:33:51.172 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-07-14T17:33:52.049 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-07-14T17:33:52.355 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-07-14T17:33:52.542 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 119/134 2023-07-14T17:33:52.735 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-14T17:33:52.910 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T17:33:52.974 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T17:33:53.090 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T17:33:53.095 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T17:33:53.096 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2023-07-14T17:33:53.096 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2023-07-14T17:33:53.096 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /dev 2023-07-14T17:33:53.096 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /run 2023-07-14T17:33:53.096 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2023-07-14T17:33:53.096 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2023-07-14T17:33:53.097 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2023-07-14T17:33:53.097 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2023-07-14T17:33:53.097 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /dev 2023-07-14T17:33:53.097 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-07-14T17:33:53.294 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T17:33:53.459 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T17:33:53.490 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-07-14T17:33:53.681 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-07-14T17:33:54.549 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-07-14T17:33:54.889 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-07-14T17:33:55.048 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T17:33:55.093 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 119/134 2023-07-14T17:33:55.199 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T17:33:55.246 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T17:33:55.289 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-14T17:33:55.447 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T17:33:55.466 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T17:33:55.466 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T17:33:55.478 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T17:33:55.640 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T17:33:55.642 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T17:33:55.658 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T17:33:55.955 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T17:33:56.097 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T17:33:56.274 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T17:33:56.276 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T17:33:56.293 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T17:33:56.566 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T17:33:56.705 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T17:33:56.713 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T17:33:56.927 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-07-14T17:33:57.080 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-07-14T17:33:57.164 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-07-14T17:33:57.380 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-07-14T17:33:57.392 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-07-14T17:33:57.392 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T17:33:57.591 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T17:33:57.592 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T17:33:57.819 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T17:33:57.857 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T17:33:57.946 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T17:33:58.062 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T17:33:58.115 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T17:33:58.132 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T17:33:58.140 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T17:33:58.256 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T17:33:58.390 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T17:33:58.524 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T17:33:58.592 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T17:33:58.593 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T17:33:58.593 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T17:33:58.593 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T17:33:58.593 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T17:33:58.593 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T17:33:58.594 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T17:33:58.594 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T17:33:58.594 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T17:33:58.594 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T17:33:58.594 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T17:33:58.594 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T17:33:58.594 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T17:33:58.595 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T17:33:58.595 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T17:33:58.595 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T17:33:58.595 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T17:33:58.595 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T17:33:58.595 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T17:33:58.595 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T17:33:58.595 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T17:33:58.596 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T17:33:58.596 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T17:33:58.596 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T17:33:58.596 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T17:33:58.596 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T17:33:58.596 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T17:33:58.597 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T17:33:58.597 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T17:33:58.597 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T17:33:58.597 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T17:33:58.597 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T17:33:58.597 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T17:33:58.597 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T17:33:58.598 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T17:33:58.598 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T17:33:58.598 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T17:33:58.604 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T17:33:58.604 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T17:33:58.604 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T17:33:58.605 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T17:33:58.605 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T17:33:58.605 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T17:33:58.605 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T17:33:58.605 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T17:33:58.605 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T17:33:58.605 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T17:33:58.606 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T17:33:58.606 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T17:33:58.606 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T17:33:58.606 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T17:33:58.606 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T17:33:58.606 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T17:33:58.606 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T17:33:58.606 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T17:33:58.607 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T17:33:58.607 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T17:33:58.607 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T17:33:58.607 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T17:33:58.607 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T17:33:58.607 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T17:33:58.608 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T17:33:58.608 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T17:33:58.608 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T17:33:58.608 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T17:33:58.608 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T17:33:58.608 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T17:33:58.608 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T17:33:58.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T17:33:58.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T17:33:58.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T17:33:58.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T17:33:58.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-07-14T17:33:58.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-07-14T17:33:58.609 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-07-14T17:33:58.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-07-14T17:33:58.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-07-14T17:33:58.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-07-14T17:33:58.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-07-14T17:33:58.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-07-14T17:33:58.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-07-14T17:33:58.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-07-14T17:33:58.610 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-07-14T17:33:58.611 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-07-14T17:33:58.611 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-07-14T17:33:58.611 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-07-14T17:33:58.611 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-07-14T17:33:58.611 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-07-14T17:33:58.611 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-07-14T17:33:58.611 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-07-14T17:33:58.611 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-07-14T17:33:58.612 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-07-14T17:33:58.612 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-07-14T17:33:58.612 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-07-14T17:33:58.612 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-07-14T17:33:58.612 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-07-14T17:33:58.612 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-07-14T17:33:58.612 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-07-14T17:33:58.613 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-07-14T17:33:58.613 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-07-14T17:33:58.613 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-07-14T17:33:58.613 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-07-14T17:33:58.613 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-07-14T17:33:58.613 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-07-14T17:33:58.613 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-07-14T17:33:58.613 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-07-14T17:33:58.614 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-07-14T17:33:58.614 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-07-14T17:33:58.614 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-07-14T17:33:58.614 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-07-14T17:33:58.614 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-07-14T17:33:58.614 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-07-14T17:33:58.614 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-07-14T17:33:58.614 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-07-14T17:33:58.615 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-07-14T17:33:58.615 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-07-14T17:33:58.615 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-07-14T17:33:58.615 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-07-14T17:33:58.615 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-07-14T17:33:58.615 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-07-14T17:33:58.615 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-14T17:33:58.616 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-07-14T17:33:58.616 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T17:33:58.616 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-07-14T17:33:58.616 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-07-14T17:33:58.616 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-07-14T17:33:58.616 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-07-14T17:33:58.616 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-07-14T17:33:58.616 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T17:33:58.617 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T17:33:58.617 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T17:33:58.617 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T17:33:58.617 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T17:33:58.617 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T17:33:58.622 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T17:33:58.724 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T17:33:58.941 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T17:33:58.969 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T17:33:59.236 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T17:33:59.391 INFO:teuthology.orchestra.run.smithi178.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T17:33:59.564 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T17:33:59.573 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T17:33:59.741 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-07-14T17:33:59.964 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-07-14T17:34:00.057 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-07-14T17:34:00.193 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T17:34:00.256 INFO:teuthology.orchestra.run.smithi178.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-07-14T17:34:00.269 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-07-14T17:34:00.269 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T17:34:00.358 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T17:34:00.449 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T17:34:00.459 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T17:34:00.459 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T17:34:00.712 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T17:34:00.840 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T17:34:00.938 INFO:teuthology.orchestra.run.smithi178.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T17:34:01.072 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T17:34:01.078 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T17:34:01.078 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:59 ago on Fri 14 Jul 2023 05:32:02 PM UTC. 2023-07-14T17:34:01.107 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T17:34:01.274 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T17:34:01.286 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T17:34:01.454 INFO:teuthology.orchestra.run.smithi178.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T17:34:01.455 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T17:34:01.455 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T17:34:01.455 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T17:34:01.455 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T17:34:01.455 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T17:34:01.455 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T17:34:01.456 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T17:34:01.456 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T17:34:01.456 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T17:34:01.456 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T17:34:01.456 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T17:34:01.456 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T17:34:01.456 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T17:34:01.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T17:34:01.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T17:34:01.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T17:34:01.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T17:34:01.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T17:34:01.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T17:34:01.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T17:34:01.457 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T17:34:01.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T17:34:01.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T17:34:01.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T17:34:01.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T17:34:01.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T17:34:01.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T17:34:01.458 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T17:34:01.465 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T17:34:01.465 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T17:34:01.465 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T17:34:01.466 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T17:34:01.466 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T17:34:01.466 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T17:34:01.466 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T17:34:01.466 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T17:34:01.466 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T17:34:01.466 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T17:34:01.467 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T17:34:01.467 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T17:34:01.467 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T17:34:01.467 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T17:34:01.467 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T17:34:01.467 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T17:34:01.467 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T17:34:01.467 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T17:34:01.468 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T17:34:01.468 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T17:34:01.468 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T17:34:01.468 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T17:34:01.468 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T17:34:01.468 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T17:34:01.468 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T17:34:01.469 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T17:34:01.469 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T17:34:01.469 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T17:34:01.469 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T17:34:01.469 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T17:34:01.469 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T17:34:01.469 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T17:34:01.469 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T17:34:01.470 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T17:34:01.470 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T17:34:01.470 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T17:34:01.470 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T17:34:01.470 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T17:34:01.470 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T17:34:01.470 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T17:34:01.470 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T17:34:01.471 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T17:34:01.471 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T17:34:01.471 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-07-14T17:34:01.471 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-07-14T17:34:01.471 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-07-14T17:34:01.471 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-07-14T17:34:01.471 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-07-14T17:34:01.471 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-07-14T17:34:01.472 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-07-14T17:34:01.472 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-07-14T17:34:01.472 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-07-14T17:34:01.472 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-07-14T17:34:01.472 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-07-14T17:34:01.472 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-07-14T17:34:01.472 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-07-14T17:34:01.473 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-07-14T17:34:01.473 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-07-14T17:34:01.473 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-07-14T17:34:01.473 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-07-14T17:34:01.473 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-07-14T17:34:01.473 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-07-14T17:34:01.473 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-07-14T17:34:01.473 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-07-14T17:34:01.474 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-07-14T17:34:01.474 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-07-14T17:34:01.474 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-07-14T17:34:01.474 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-07-14T17:34:01.474 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-07-14T17:34:01.474 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-07-14T17:34:01.474 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-07-14T17:34:01.475 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-07-14T17:34:01.475 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-07-14T17:34:01.475 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-07-14T17:34:01.475 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-07-14T17:34:01.475 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-07-14T17:34:01.475 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-07-14T17:34:01.475 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-07-14T17:34:01.475 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-07-14T17:34:01.476 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-07-14T17:34:01.476 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-07-14T17:34:01.476 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-07-14T17:34:01.476 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-07-14T17:34:01.476 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-07-14T17:34:01.476 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-07-14T17:34:01.476 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-07-14T17:34:01.476 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-07-14T17:34:01.477 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-07-14T17:34:01.477 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-07-14T17:34:01.477 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-07-14T17:34:01.477 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-07-14T17:34:01.477 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-14T17:34:01.477 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-07-14T17:34:01.477 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T17:34:01.478 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-07-14T17:34:01.478 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-07-14T17:34:01.478 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-07-14T17:34:01.478 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-07-14T17:34:01.478 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-07-14T17:34:01.478 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T17:34:01.478 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T17:34:01.478 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T17:34:01.479 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T17:34:01.479 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T17:34:01.479 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T17:34:02.584 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T17:34:02.821 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T17:34:02.855 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T17:34:03.007 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T17:34:03.050 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T17:34:03.232 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T17:34:03.584 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T17:34:03.790 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T17:34:04.127 INFO:teuthology.orchestra.run.smithi178.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T17:34:04.128 INFO:teuthology.orchestra.run.smithi178.stdout:Last metadata expiration check: 0:02:01 ago on Fri 14 Jul 2023 05:32:03 PM UTC. 2023-07-14T17:34:08.345 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T17:34:09.143 INFO:teuthology.orchestra.run.smithi088.stdout:Installed products updated. 2023-07-14T17:34:09.257 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T17:34:09.357 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:34:09.357 INFO:teuthology.orchestra.run.smithi088.stdout:Upgraded: 2023-07-14T17:34:09.357 INFO:teuthology.orchestra.run.smithi088.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T17:34:09.357 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.358 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.358 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-07-14T17:34:09.358 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T17:34:09.358 INFO:teuthology.orchestra.run.smithi088.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T17:34:09.358 INFO:teuthology.orchestra.run.smithi088.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T17:34:09.358 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.359 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.359 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.359 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.359 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:09.359 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.359 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.359 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.359 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:09.360 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:09.360 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:09.360 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:09.360 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:09.360 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.360 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.360 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:09.361 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.361 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.361 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.361 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:09.361 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:09.361 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.361 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:09.361 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T17:34:09.362 INFO:teuthology.orchestra.run.smithi088.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T17:34:09.362 INFO:teuthology.orchestra.run.smithi088.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T17:34:09.362 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T17:34:09.362 INFO:teuthology.orchestra.run.smithi088.stdout: jq-1.6-3.el8.x86_64 2023-07-14T17:34:09.362 INFO:teuthology.orchestra.run.smithi088.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T17:34:09.362 INFO:teuthology.orchestra.run.smithi088.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T17:34:09.362 INFO:teuthology.orchestra.run.smithi088.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T17:34:09.362 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.363 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.363 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.363 INFO:teuthology.orchestra.run.smithi088.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T17:34:09.363 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T17:34:09.363 INFO:teuthology.orchestra.run.smithi088.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T17:34:09.363 INFO:teuthology.orchestra.run.smithi088.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T17:34:09.363 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T17:34:09.364 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T17:34:09.364 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.364 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.364 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T17:34:09.364 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.364 INFO:teuthology.orchestra.run.smithi088.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T17:34:09.364 INFO:teuthology.orchestra.run.smithi088.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T17:34:09.364 INFO:teuthology.orchestra.run.smithi088.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T17:34:09.365 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T17:34:09.365 INFO:teuthology.orchestra.run.smithi088.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T17:34:09.365 INFO:teuthology.orchestra.run.smithi088.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T17:34:09.365 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T17:34:09.365 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T17:34:09.365 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T17:34:09.365 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T17:34:09.365 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T17:34:09.366 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T17:34:09.366 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T17:34:09.366 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T17:34:09.366 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T17:34:09.366 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T17:34:09.366 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T17:34:09.366 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T17:34:09.367 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T17:34:09.367 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T17:34:09.367 INFO:teuthology.orchestra.run.smithi088.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T17:34:09.367 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T17:34:09.367 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.367 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.367 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.367 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T17:34:09.368 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T17:34:09.368 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T17:34:09.368 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T17:34:09.368 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T17:34:09.368 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T17:34:09.368 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T17:34:09.368 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T17:34:09.368 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T17:34:09.369 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T17:34:09.369 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T17:34:09.369 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T17:34:09.369 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T17:34:09.369 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T17:34:09.369 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T17:34:09.369 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T17:34:09.369 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T17:34:09.370 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T17:34:09.370 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T17:34:09.370 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T17:34:09.370 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T17:34:09.370 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T17:34:09.370 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T17:34:09.370 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T17:34:09.370 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T17:34:09.371 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T17:34:09.371 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T17:34:09.371 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T17:34:09.371 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T17:34:09.371 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.371 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.371 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T17:34:09.372 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T17:34:09.372 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.372 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T17:34:09.372 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T17:34:09.372 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T17:34:09.372 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T17:34:09.372 INFO:teuthology.orchestra.run.smithi088.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T17:34:09.372 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T17:34:09.373 INFO:teuthology.orchestra.run.smithi088.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T17:34:09.373 INFO:teuthology.orchestra.run.smithi088.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T17:34:09.373 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T17:34:09.373 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T17:34:09.373 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T17:34:09.373 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T17:34:09.373 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T17:34:09.373 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.374 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.374 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:09.374 INFO:teuthology.orchestra.run.smithi088.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T17:34:09.374 INFO:teuthology.orchestra.run.smithi088.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T17:34:09.374 INFO:teuthology.orchestra.run.smithi088.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T17:34:09.374 INFO:teuthology.orchestra.run.smithi088.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T17:34:09.374 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T17:34:09.375 INFO:teuthology.orchestra.run.smithi088.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T17:34:09.375 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:34:09.375 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-07-14T17:34:09.540 DEBUG:teuthology.parallel:result is None 2023-07-14T17:34:09.604 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-14T17:34:09.825 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 117/134 2023-07-14T17:34:10.012 INFO:teuthology.orchestra.run.smithi178.stdout:Installed products updated. 2023-07-14T17:34:10.049 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-14T17:34:10.224 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-07-14T17:34:10.224 INFO:teuthology.orchestra.run.smithi178.stdout:Upgraded: 2023-07-14T17:34:10.224 INFO:teuthology.orchestra.run.smithi178.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T17:34:10.225 INFO:teuthology.orchestra.run.smithi178.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.225 INFO:teuthology.orchestra.run.smithi178.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.225 INFO:teuthology.orchestra.run.smithi178.stdout:Installed: 2023-07-14T17:34:10.225 INFO:teuthology.orchestra.run.smithi178.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T17:34:10.225 INFO:teuthology.orchestra.run.smithi178.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T17:34:10.225 INFO:teuthology.orchestra.run.smithi178.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T17:34:10.225 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.226 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.226 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.226 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.226 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:10.226 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.226 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.226 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.226 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:10.227 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:10.227 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:10.227 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:10.227 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:10.227 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.227 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.227 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:10.228 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.228 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.228 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.228 INFO:teuthology.orchestra.run.smithi178.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:10.228 INFO:teuthology.orchestra.run.smithi178.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:10.228 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.228 INFO:teuthology.orchestra.run.smithi178.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:10.228 INFO:teuthology.orchestra.run.smithi178.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T17:34:10.229 INFO:teuthology.orchestra.run.smithi178.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T17:34:10.229 INFO:teuthology.orchestra.run.smithi178.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T17:34:10.229 INFO:teuthology.orchestra.run.smithi178.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T17:34:10.229 INFO:teuthology.orchestra.run.smithi178.stdout: jq-1.6-3.el8.x86_64 2023-07-14T17:34:10.229 INFO:teuthology.orchestra.run.smithi178.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T17:34:10.229 INFO:teuthology.orchestra.run.smithi178.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T17:34:10.229 INFO:teuthology.orchestra.run.smithi178.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T17:34:10.230 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.230 INFO:teuthology.orchestra.run.smithi178.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.230 INFO:teuthology.orchestra.run.smithi178.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.230 INFO:teuthology.orchestra.run.smithi178.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T17:34:10.230 INFO:teuthology.orchestra.run.smithi178.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T17:34:10.230 INFO:teuthology.orchestra.run.smithi178.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T17:34:10.230 INFO:teuthology.orchestra.run.smithi178.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T17:34:10.230 INFO:teuthology.orchestra.run.smithi178.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T17:34:10.231 INFO:teuthology.orchestra.run.smithi178.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T17:34:10.231 INFO:teuthology.orchestra.run.smithi178.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.231 INFO:teuthology.orchestra.run.smithi178.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.231 INFO:teuthology.orchestra.run.smithi178.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T17:34:10.231 INFO:teuthology.orchestra.run.smithi178.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.231 INFO:teuthology.orchestra.run.smithi178.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T17:34:10.231 INFO:teuthology.orchestra.run.smithi178.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T17:34:10.231 INFO:teuthology.orchestra.run.smithi178.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T17:34:10.232 INFO:teuthology.orchestra.run.smithi178.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T17:34:10.232 INFO:teuthology.orchestra.run.smithi178.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T17:34:10.232 INFO:teuthology.orchestra.run.smithi178.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T17:34:10.232 INFO:teuthology.orchestra.run.smithi178.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T17:34:10.232 INFO:teuthology.orchestra.run.smithi178.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T17:34:10.232 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T17:34:10.232 INFO:teuthology.orchestra.run.smithi178.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T17:34:10.233 INFO:teuthology.orchestra.run.smithi178.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T17:34:10.233 INFO:teuthology.orchestra.run.smithi178.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T17:34:10.233 INFO:teuthology.orchestra.run.smithi178.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T17:34:10.233 INFO:teuthology.orchestra.run.smithi178.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T17:34:10.233 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T17:34:10.233 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T17:34:10.233 INFO:teuthology.orchestra.run.smithi178.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T17:34:10.233 INFO:teuthology.orchestra.run.smithi178.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T17:34:10.234 INFO:teuthology.orchestra.run.smithi178.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T17:34:10.234 INFO:teuthology.orchestra.run.smithi178.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T17:34:10.234 INFO:teuthology.orchestra.run.smithi178.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T17:34:10.234 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T17:34:10.234 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.234 INFO:teuthology.orchestra.run.smithi178.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.234 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.235 INFO:teuthology.orchestra.run.smithi178.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T17:34:10.235 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T17:34:10.235 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T17:34:10.235 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T17:34:10.235 INFO:teuthology.orchestra.run.smithi178.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T17:34:10.235 INFO:teuthology.orchestra.run.smithi178.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T17:34:10.235 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T17:34:10.235 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T17:34:10.236 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T17:34:10.236 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T17:34:10.236 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T17:34:10.236 INFO:teuthology.orchestra.run.smithi178.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T17:34:10.236 INFO:teuthology.orchestra.run.smithi178.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T17:34:10.236 INFO:teuthology.orchestra.run.smithi178.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T17:34:10.236 INFO:teuthology.orchestra.run.smithi178.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T17:34:10.236 INFO:teuthology.orchestra.run.smithi178.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T17:34:10.237 INFO:teuthology.orchestra.run.smithi178.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T17:34:10.237 INFO:teuthology.orchestra.run.smithi178.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T17:34:10.237 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T17:34:10.237 INFO:teuthology.orchestra.run.smithi178.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T17:34:10.237 INFO:teuthology.orchestra.run.smithi178.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T17:34:10.237 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T17:34:10.237 INFO:teuthology.orchestra.run.smithi178.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T17:34:10.238 INFO:teuthology.orchestra.run.smithi178.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T17:34:10.238 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T17:34:10.238 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T17:34:10.238 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T17:34:10.238 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T17:34:10.238 INFO:teuthology.orchestra.run.smithi178.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T17:34:10.238 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.238 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.239 INFO:teuthology.orchestra.run.smithi178.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T17:34:10.239 INFO:teuthology.orchestra.run.smithi178.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T17:34:10.239 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.239 INFO:teuthology.orchestra.run.smithi178.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T17:34:10.240 INFO:teuthology.orchestra.run.smithi178.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T17:34:10.240 INFO:teuthology.orchestra.run.smithi178.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T17:34:10.240 INFO:teuthology.orchestra.run.smithi178.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T17:34:10.240 INFO:teuthology.orchestra.run.smithi178.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T17:34:10.240 INFO:teuthology.orchestra.run.smithi178.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T17:34:10.240 INFO:teuthology.orchestra.run.smithi178.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T17:34:10.240 INFO:teuthology.orchestra.run.smithi178.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T17:34:10.241 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T17:34:10.241 INFO:teuthology.orchestra.run.smithi178.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T17:34:10.241 INFO:teuthology.orchestra.run.smithi178.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T17:34:10.241 INFO:teuthology.orchestra.run.smithi178.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T17:34:10.241 INFO:teuthology.orchestra.run.smithi178.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T17:34:10.241 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.241 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.241 INFO:teuthology.orchestra.run.smithi178.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:10.242 INFO:teuthology.orchestra.run.smithi178.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T17:34:10.242 INFO:teuthology.orchestra.run.smithi178.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T17:34:10.242 INFO:teuthology.orchestra.run.smithi178.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T17:34:10.242 INFO:teuthology.orchestra.run.smithi178.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T17:34:10.242 INFO:teuthology.orchestra.run.smithi178.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T17:34:10.243 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T17:34:10.243 INFO:teuthology.orchestra.run.smithi178.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T17:34:10.243 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-07-14T17:34:10.243 INFO:teuthology.orchestra.run.smithi178.stdout:Complete! 2023-07-14T17:34:10.406 DEBUG:teuthology.parallel:result is None 2023-07-14T17:34:10.437 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T17:34:10.693 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-14T17:34:10.868 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T17:34:11.065 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T17:34:13.207 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T17:34:13.480 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T17:34:13.499 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T17:34:13.730 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T17:34:13.996 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T17:34:14.345 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T17:34:14.748 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T17:34:14.956 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T17:34:15.040 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T17:34:15.246 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T17:34:15.258 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T17:34:15.437 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T17:34:15.446 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T17:34:15.556 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-14T17:34:15.557 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T17:34:15.757 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T17:34:15.758 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T17:34:16.035 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T17:34:16.270 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T17:34:16.793 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T17:34:16.793 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T17:34:16.794 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T17:34:16.794 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T17:34:16.794 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T17:34:16.798 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T17:34:16.799 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T17:34:16.799 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T17:34:16.799 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T17:34:16.799 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T17:34:16.799 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T17:34:16.799 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T17:34:16.799 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T17:34:16.800 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T17:34:16.800 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T17:34:16.800 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T17:34:16.800 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T17:34:16.800 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T17:34:16.800 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T17:34:16.800 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T17:34:16.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T17:34:16.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T17:34:16.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T17:34:16.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T17:34:16.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T17:34:16.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T17:34:16.801 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T17:34:16.802 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T17:34:16.802 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T17:34:16.802 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T17:34:16.802 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T17:34:16.802 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T17:34:16.802 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T17:34:16.802 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T17:34:16.803 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T17:34:16.803 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T17:34:16.803 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T17:34:16.804 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T17:34:16.804 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T17:34:16.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T17:34:16.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T17:34:16.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T17:34:16.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T17:34:16.806 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T17:34:16.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T17:34:16.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T17:34:16.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T17:34:16.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T17:34:16.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T17:34:16.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T17:34:16.807 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T17:34:16.808 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T17:34:16.808 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T17:34:16.813 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T17:34:16.814 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T17:34:16.814 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T17:34:16.814 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T17:34:16.814 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T17:34:16.814 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T17:34:16.815 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T17:34:16.815 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T17:34:16.815 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T17:34:16.815 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T17:34:16.815 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T17:34:16.815 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T17:34:16.816 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T17:34:16.816 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T17:34:16.816 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T17:34:16.816 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T17:34:16.816 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T17:34:16.816 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T17:34:16.817 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T17:34:16.817 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-14T17:34:16.817 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-14T17:34:16.817 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-14T17:34:16.817 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-14T17:34:16.817 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-14T17:34:16.817 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-14T17:34:16.818 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-14T17:34:16.818 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-14T17:34:16.818 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-14T17:34:16.818 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-14T17:34:16.818 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-14T17:34:16.818 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-14T17:34:16.818 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-14T17:34:16.819 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-14T17:34:16.819 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-14T17:34:16.819 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-14T17:34:16.819 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-14T17:34:16.819 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-14T17:34:16.819 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-14T17:34:16.819 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-14T17:34:16.820 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-14T17:34:16.820 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-14T17:34:16.820 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-14T17:34:16.821 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-14T17:34:16.821 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-14T17:34:16.821 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-14T17:34:16.821 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-14T17:34:16.821 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-14T17:34:16.821 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-14T17:34:16.821 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-14T17:34:16.822 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-14T17:34:16.822 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-14T17:34:16.822 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-14T17:34:16.822 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-14T17:34:16.822 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-14T17:34:16.822 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-14T17:34:16.822 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-14T17:34:16.823 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-14T17:34:16.823 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-14T17:34:16.823 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-14T17:34:16.823 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-14T17:34:16.823 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-14T17:34:16.823 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-14T17:34:16.823 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-14T17:34:16.824 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-14T17:34:16.824 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-14T17:34:16.824 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-14T17:34:16.824 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-14T17:34:16.824 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-14T17:34:16.824 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-14T17:34:16.825 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-14T17:34:16.825 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-14T17:34:16.825 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-14T17:34:16.825 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-14T17:34:16.825 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-14T17:34:16.825 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-14T17:34:16.825 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-14T17:34:16.826 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T17:34:16.827 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T17:34:16.827 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T17:34:16.827 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T17:34:16.827 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T17:34:19.343 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T17:34:19.343 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:00 ago on Fri 14 Jul 2023 05:32:19 PM UTC. 2023-07-14T17:34:29.919 INFO:teuthology.orchestra.run.smithi107.stdout:Installed products updated. 2023-07-14T17:34:30.133 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-07-14T17:34:30.134 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2023-07-14T17:34:30.134 INFO:teuthology.orchestra.run.smithi107.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T17:34:30.134 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.134 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.134 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2023-07-14T17:34:30.134 INFO:teuthology.orchestra.run.smithi107.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T17:34:30.135 INFO:teuthology.orchestra.run.smithi107.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T17:34:30.135 INFO:teuthology.orchestra.run.smithi107.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T17:34:30.135 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.135 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.135 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.135 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.135 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:30.135 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.136 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.136 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.136 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:30.136 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:30.136 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:30.136 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:30.136 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:30.137 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.137 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.137 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:30.137 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.137 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.137 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.137 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:30.137 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:30.138 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.138 INFO:teuthology.orchestra.run.smithi107.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T17:34:30.138 INFO:teuthology.orchestra.run.smithi107.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T17:34:30.138 INFO:teuthology.orchestra.run.smithi107.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T17:34:30.138 INFO:teuthology.orchestra.run.smithi107.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T17:34:30.138 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T17:34:30.138 INFO:teuthology.orchestra.run.smithi107.stdout: jq-1.6-3.el8.x86_64 2023-07-14T17:34:30.139 INFO:teuthology.orchestra.run.smithi107.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T17:34:30.139 INFO:teuthology.orchestra.run.smithi107.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T17:34:30.139 INFO:teuthology.orchestra.run.smithi107.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T17:34:30.139 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.139 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.139 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.139 INFO:teuthology.orchestra.run.smithi107.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T17:34:30.140 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T17:34:30.140 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T17:34:30.140 INFO:teuthology.orchestra.run.smithi107.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T17:34:30.140 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T17:34:30.140 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T17:34:30.141 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.141 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.141 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T17:34:30.141 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.141 INFO:teuthology.orchestra.run.smithi107.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T17:34:30.141 INFO:teuthology.orchestra.run.smithi107.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T17:34:30.141 INFO:teuthology.orchestra.run.smithi107.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T17:34:30.142 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T17:34:30.142 INFO:teuthology.orchestra.run.smithi107.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T17:34:30.142 INFO:teuthology.orchestra.run.smithi107.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T17:34:30.142 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T17:34:30.142 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T17:34:30.142 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T17:34:30.142 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T17:34:30.142 INFO:teuthology.orchestra.run.smithi107.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T17:34:30.143 INFO:teuthology.orchestra.run.smithi107.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T17:34:30.143 INFO:teuthology.orchestra.run.smithi107.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T17:34:30.143 INFO:teuthology.orchestra.run.smithi107.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T17:34:30.143 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T17:34:30.143 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T17:34:30.143 INFO:teuthology.orchestra.run.smithi107.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T17:34:30.143 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T17:34:30.144 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T17:34:30.144 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T17:34:30.144 INFO:teuthology.orchestra.run.smithi107.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T17:34:30.144 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T17:34:30.144 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.144 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.144 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T17:34:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T17:34:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T17:34:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T17:34:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T17:34:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T17:34:30.145 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T17:34:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T17:34:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T17:34:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T17:34:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T17:34:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T17:34:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T17:34:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T17:34:30.146 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T17:34:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T17:34:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T17:34:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T17:34:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T17:34:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T17:34:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T17:34:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T17:34:30.147 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T17:34:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T17:34:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T17:34:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T17:34:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T17:34:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T17:34:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T17:34:30.148 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T17:34:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T17:34:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T17:34:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T17:34:30.149 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T17:34:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T17:34:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T17:34:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T17:34:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T17:34:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T17:34:30.150 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T17:34:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T17:34:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T17:34:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T17:34:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T17:34:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.151 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T17:34:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T17:34:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T17:34:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T17:34:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T17:34:30.152 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T17:34:30.153 INFO:teuthology.orchestra.run.smithi107.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T17:34:30.153 INFO:teuthology.orchestra.run.smithi107.stdout: 2023-07-14T17:34:30.153 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2023-07-14T17:34:30.360 DEBUG:teuthology.parallel:result is None 2023-07-14T17:34:30.361 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T17:34:35.601 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T17:34:35.626 INFO:teuthology.orchestra.run.smithi088.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T17:34:35.627 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T17:34:35.627 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T17:34:35.628 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T17:34:35.890 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T17:34:35.918 INFO:teuthology.orchestra.run.smithi107.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T17:34:35.919 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T17:34:35.919 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T17:34:35.919 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T17:34:36.142 DEBUG:teuthology.orchestra.run.smithi178:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T17:34:36.171 INFO:teuthology.orchestra.run.smithi178.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T17:34:36.171 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T17:34:36.171 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T17:34:36.172 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-14T17:34:36.172 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:34:36.173 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T17:34:36.220 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-07-14T17:34:36.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T17:34:36.267 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-07-14T17:34:36.267 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T17:34:36.314 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-14T17:34:36.315 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:34:36.315 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T17:34:36.346 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T17:34:36.416 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-07-14T17:34:36.416 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T17:34:36.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T17:34:36.516 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-07-14T17:34:36.517 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T17:34:36.548 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T17:34:36.620 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-14T17:34:36.621 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:34:36.621 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T17:34:36.651 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T17:34:36.721 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-07-14T17:34:36.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T17:34:36.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T17:34:36.820 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-07-14T17:34:36.820 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T17:34:36.852 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T17:34:36.923 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-14T17:34:36.923 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:34:36.923 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T17:34:36.953 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T17:34:37.022 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-07-14T17:34:37.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T17:34:37.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T17:34:37.120 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-07-14T17:34:37.121 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T17:34:37.152 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T17:34:37.222 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-14T17:34:37.312 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2023-07-14T17:34:37.313 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T17:34:37.324 INFO:tasks.cephadm:Cluster fsid is b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:34:37.325 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-14T17:34:37.325 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.88', 'mon.b': '172.21.15.107', 'mon.c': '172.21.15.178'} 2023-07-14T17:34:37.325 INFO:tasks.cephadm:First mon is mon.a on smithi088 2023-07-14T17:34:37.326 INFO:tasks.cephadm:First mgr is x 2023-07-14T17:34:37.326 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-14T17:34:37.326 DEBUG:teuthology.orchestra.run.smithi088:> sudo hostname $(hostname -s) 2023-07-14T17:34:37.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname $(hostname -s) 2023-07-14T17:34:37.389 DEBUG:teuthology.orchestra.run.smithi178:> sudo hostname $(hostname -s) 2023-07-14T17:34:37.420 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-07-14T17:34:37.420 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T17:34:42.804 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71431/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-4940-g81c9387d', 'node_name': '172.21.4.65+confusa03', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-4940.g81c9387d'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-14 16:50:26.249583', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20230714.134335', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/', 'archs': ['source', 'x86_64', 'arm64'], 'distro': 'centos'}] 2023-07-14T17:34:48.231 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-pdonnell-testing-20230714.134335, sha1 81c9387d2e7576da065747ec9c16d1c7d69016ec from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T17:34:48.233 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm 2023-07-14T17:34:48.233 DEBUG:teuthology.orchestra.run.smithi088:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T17:34:49.221 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 17:34 /home/ubuntu/cephtest/cephadm 2023-07-14T17:34:49.222 DEBUG:teuthology.orchestra.run.smithi107:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T17:34:50.143 INFO:teuthology.orchestra.run.smithi107.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 17:34 /home/ubuntu/cephtest/cephadm 2023-07-14T17:34:50.143 DEBUG:teuthology.orchestra.run.smithi178:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T17:34:51.116 INFO:teuthology.orchestra.run.smithi178.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 17:34 /home/ubuntu/cephtest/cephadm 2023-07-14T17:34:51.117 DEBUG:teuthology.orchestra.run.smithi088:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T17:34:51.138 DEBUG:teuthology.orchestra.run.smithi107:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T17:34:51.160 DEBUG:teuthology.orchestra.run.smithi178:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T17:34:51.195 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec on all hosts... 2023-07-14T17:34:51.195 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T17:34:51.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T17:34:51.202 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T17:34:52.516 INFO:teuthology.orchestra.run.smithi178.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T17:34:52.539 INFO:teuthology.orchestra.run.smithi088.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T17:34:52.595 INFO:teuthology.orchestra.run.smithi107.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T17:35:30.270 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-07-14T17:35:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T17:35:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T17:35:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: "repo_digests": [ 2023-07-14T17:35:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T17:35:30.271 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-07-14T17:35:30.271 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-07-14T17:35:40.803 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2023-07-14T17:35:40.804 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T17:35:40.804 INFO:teuthology.orchestra.run.smithi107.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T17:35:40.804 INFO:teuthology.orchestra.run.smithi107.stdout: "repo_digests": [ 2023-07-14T17:35:40.804 INFO:teuthology.orchestra.run.smithi107.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T17:35:40.804 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2023-07-14T17:35:40.804 INFO:teuthology.orchestra.run.smithi107.stdout:} 2023-07-14T17:35:41.009 INFO:teuthology.orchestra.run.smithi178.stdout:{ 2023-07-14T17:35:41.009 INFO:teuthology.orchestra.run.smithi178.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T17:35:41.010 INFO:teuthology.orchestra.run.smithi178.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T17:35:41.010 INFO:teuthology.orchestra.run.smithi178.stdout: "repo_digests": [ 2023-07-14T17:35:41.010 INFO:teuthology.orchestra.run.smithi178.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T17:35:41.010 INFO:teuthology.orchestra.run.smithi178.stdout: ] 2023-07-14T17:35:41.010 INFO:teuthology.orchestra.run.smithi178.stdout:} 2023-07-14T17:35:41.032 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /etc/ceph 2023-07-14T17:35:41.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph 2023-07-14T17:35:41.097 DEBUG:teuthology.orchestra.run.smithi178:> sudo mkdir -p /etc/ceph 2023-07-14T17:35:41.127 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 777 /etc/ceph 2023-07-14T17:35:41.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /etc/ceph 2023-07-14T17:35:41.184 DEBUG:teuthology.orchestra.run.smithi178:> sudo chmod 777 /etc/ceph 2023-07-14T17:35:41.211 INFO:tasks.cephadm:Writing seed config... 2023-07-14T17:35:41.212 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-07-14T17:35:41.213 INFO:tasks.cephadm: override: [client] debug client = 20 2023-07-14T17:35:41.213 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-07-14T17:35:41.213 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-07-14T17:35:41.213 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-07-14T17:35:41.214 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-07-14T17:35:41.214 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-07-14T17:35:41.214 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-07-14T17:35:41.214 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-07-14T17:35:41.215 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-07-14T17:35:41.215 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-07-14T17:35:41.215 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-07-14T17:35:41.215 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-07-14T17:35:41.215 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-07-14T17:35:41.215 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-07-14T17:35:41.216 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-07-14T17:35:41.216 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2023-07-14T17:35:41.216 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-07-14T17:35:41.216 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-07-14T17:35:41.216 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-07-14T17:35:41.217 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-07-14T17:35:41.217 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-14T17:35:41.217 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-14T17:35:41.217 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-14T17:35:41.217 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-14T17:35:41.218 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-14T17:35:41.218 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-07-14T17:35:41.218 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-07-14T17:35:41.218 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-07-14T17:35:41.218 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-07-14T17:35:41.219 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-07-14T17:35:41.219 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-14T17:35:41.219 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-14T17:35:41.219 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-07-14T17:35:41.219 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-07-14T17:35:41.220 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-07-14T17:35:41.220 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-07-14T17:35:41.220 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-07-14T17:35:41.220 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-07-14T17:35:41.220 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-07-14T17:35:41.221 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-07-14T17:35:41.221 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2023-07-14T17:35:41.222 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:35:41.222 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-14T17:35:41.269 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = b462a556-226c-11ee-9b33-001a4aab830c [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-07-14T17:35:41.270 DEBUG:teuthology.orchestra.run.smithi088:mon.a> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mon.a.service 2023-07-14T17:35:41.312 DEBUG:teuthology.orchestra.run.smithi088:mgr.x> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mgr.x.service 2023-07-14T17:35:41.339 INFO:journalctl@ceph.mon.a.smithi088.stdout:-- Logs begin at Fri 2023-07-14 17:14:35 UTC. -- 2023-07-14T17:35:41.340 INFO:tasks.cephadm:Bootstrapping... 2023-07-14T17:35:41.340 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec -v bootstrap --fsid b462a556-226c-11ee-9b33-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.88 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-14T17:35:41.366 INFO:journalctl@ceph.mgr.x.smithi088.stdout:-- Logs begin at Fri 2023-07-14 17:14:35 UTC. -- 2023-07-14T17:35:41.500 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-07-14T17:35:41.500 INFO:teuthology.orchestra.run.smithi088.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec', '-v', 'bootstrap', '--fsid', 'b462a556-226c-11ee-9b33-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.88', '--skip-admin-label'] 2023-07-14T17:35:41.645 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stdout 4.1.1 2023-07-14T17:35:41.646 INFO:teuthology.orchestra.run.smithi088.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-07-14T17:35:41.646 INFO:teuthology.orchestra.run.smithi088.stdout:Verifying podman|docker is present... 2023-07-14T17:35:41.789 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stdout 4.1.1 2023-07-14T17:35:41.789 INFO:teuthology.orchestra.run.smithi088.stdout:Verifying lvm2 is present... 2023-07-14T17:35:41.789 INFO:teuthology.orchestra.run.smithi088.stdout:Verifying time synchronization is in place... 2023-07-14T17:35:41.795 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-14T17:35:41.795 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-14T17:35:41.801 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-14T17:35:41.801 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout inactive 2023-07-14T17:35:41.807 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout enabled 2023-07-14T17:35:41.813 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout active 2023-07-14T17:35:41.813 INFO:teuthology.orchestra.run.smithi088.stdout:Unit chronyd.service is enabled and running 2023-07-14T17:35:41.814 INFO:teuthology.orchestra.run.smithi088.stdout:Repeating the final host check... 2023-07-14T17:35:41.959 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stdout 4.1.1 2023-07-14T17:35:41.959 INFO:teuthology.orchestra.run.smithi088.stdout:podman (/bin/podman) version 4.1.1 is present 2023-07-14T17:35:41.959 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl is present 2023-07-14T17:35:41.960 INFO:teuthology.orchestra.run.smithi088.stdout:lvcreate is present 2023-07-14T17:35:41.965 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-14T17:35:41.965 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-14T17:35:41.971 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-14T17:35:41.971 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout inactive 2023-07-14T17:35:41.977 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout enabled 2023-07-14T17:35:41.983 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout active 2023-07-14T17:35:41.983 INFO:teuthology.orchestra.run.smithi088.stdout:Unit chronyd.service is enabled and running 2023-07-14T17:35:41.983 INFO:teuthology.orchestra.run.smithi088.stdout:Host looks OK 2023-07-14T17:35:41.983 INFO:teuthology.orchestra.run.smithi088.stdout:Cluster fsid: b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:35:41.983 INFO:teuthology.orchestra.run.smithi088.stdout:Acquiring lock 140641150319080 on /run/cephadm/b462a556-226c-11ee-9b33-001a4aab830c.lock 2023-07-14T17:35:41.984 INFO:teuthology.orchestra.run.smithi088.stdout:Lock 140641150319080 acquired on /run/cephadm/b462a556-226c-11ee-9b33-001a4aab830c.lock 2023-07-14T17:35:41.984 INFO:teuthology.orchestra.run.smithi088.stdout:Verifying IP 172.21.15.88 port 3300 ... 2023-07-14T17:35:41.984 INFO:teuthology.orchestra.run.smithi088.stdout:Verifying IP 172.21.15.88 port 6789 ... 2023-07-14T17:35:41.984 INFO:teuthology.orchestra.run.smithi088.stdout:Base mon IP(s) is [172.21.15.88:3300, 172.21.15.88:6789], mon addrv is [v2:172.21.15.88:3300,v1:172.21.15.88:6789] 2023-07-14T17:35:41.988 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2023-07-14T17:35:41.988 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.88 metric 100 2023-07-14T17:35:41.990 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-07-14T17:35:41.991 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2023-07-14T17:35:41.991 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2023-07-14T17:35:41.993 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-14T17:35:41.993 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-07-14T17:35:41.993 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-14T17:35:41.994 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-07-14T17:35:41.994 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout inet6 fe80::4893:55f2:c841:20/64 scope link noprefixroute 2023-07-14T17:35:41.994 INFO:teuthology.orchestra.run.smithi088.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-14T17:35:41.994 INFO:teuthology.orchestra.run.smithi088.stdout:Mon IP `172.21.15.88` is in CIDR network `172.21.0.0/20` 2023-07-14T17:35:41.995 INFO:teuthology.orchestra.run.smithi088.stdout:Mon IP `172.21.15.88` is in CIDR network `172.21.0.0/20` 2023-07-14T17:35:41.995 INFO:teuthology.orchestra.run.smithi088.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-07-14T17:35:41.995 INFO:teuthology.orchestra.run.smithi088.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-14T17:35:41.996 INFO:teuthology.orchestra.run.smithi088.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T17:35:42.643 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stdout 219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705 2023-07-14T17:35:42.643 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T17:35:42.643 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stderr Getting image source signatures 2023-07-14T17:35:42.643 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stderr Copying blob sha256:0e454b422ba90e2ad13e636c40610401086a3576fe973b0460943839fe5f2508 2023-07-14T17:35:42.643 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stderr Copying blob sha256:6022c611d474f2b306ddd1edfc8360123ece5266119096f25fde843d7967e636 2023-07-14T17:35:42.643 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stderr Copying config sha256:219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705 2023-07-14T17:35:42.644 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stderr Writing manifest to image destination 2023-07-14T17:35:42.644 INFO:teuthology.orchestra.run.smithi088.stdout:/bin/podman: stderr Storing signatures 2023-07-14T17:35:43.396 INFO:teuthology.orchestra.run.smithi088.stdout:ceph: stdout ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev) 2023-07-14T17:35:43.397 INFO:teuthology.orchestra.run.smithi088.stdout:Ceph version: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev) 2023-07-14T17:35:43.397 INFO:teuthology.orchestra.run.smithi088.stdout:Extracting ceph user uid/gid from container image... 2023-07-14T17:35:44.146 INFO:teuthology.orchestra.run.smithi088.stdout:stat: stdout 167 167 2023-07-14T17:35:44.146 INFO:teuthology.orchestra.run.smithi088.stdout:Creating initial keys... 2023-07-14T17:35:44.898 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph-authtool: stdout AQBwh7Fk5rsxHxAAAi7lBE56D406g6r5BiI15A== 2023-07-14T17:35:45.652 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph-authtool: stdout AQBxh7FkrI3UDhAAHSX5251LHjj1gfVstS+9Rg== 2023-07-14T17:35:46.296 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph-authtool: stdout AQByh7FktsWlABAAzqcyLtb1RGSbMu0NPEQq0Q== 2023-07-14T17:35:46.296 INFO:teuthology.orchestra.run.smithi088.stdout:Creating initial monmap... 2023-07-14T17:35:47.046 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-14T17:35:47.046 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-07-14T17:35:47.046 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:35:47.046 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-14T17:35:47.046 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool for a [v2:172.21.15.88:3300,v1:172.21.15.88:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-14T17:35:47.047 INFO:teuthology.orchestra.run.smithi088.stdout:setting min_mon_release = pacific 2023-07-14T17:35:47.047 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/monmaptool: set fsid to b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:35:47.047 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-14T17:35:47.047 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:35:47.047 INFO:teuthology.orchestra.run.smithi088.stdout:Creating mon... 2023-07-14T17:35:47.797 INFO:teuthology.orchestra.run.smithi088.stdout:create mon.a on 2023-07-14T17:35:47.915 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-07-14T17:35:48.005 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-07-14T17:35:48.118 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-b462a556-226c-11ee-9b33-001a4aab830c.target → /etc/systemd/system/ceph-b462a556-226c-11ee-9b33-001a4aab830c.target. 2023-07-14T17:35:48.119 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-b462a556-226c-11ee-9b33-001a4aab830c.target → /etc/systemd/system/ceph-b462a556-226c-11ee-9b33-001a4aab830c.target. 2023-07-14T17:35:48.279 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-b462a556-226c-11ee-9b33-001a4aab830c@mon.a 2023-07-14T17:35:48.279 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Failed to reset failed state of unit ceph-b462a556-226c-11ee-9b33-001a4aab830c@mon.a.service: Unit ceph-b462a556-226c-11ee-9b33-001a4aab830c@mon.a.service not loaded. 2023-07-14T17:35:48.363 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-b462a556-226c-11ee-9b33-001a4aab830c.target.wants/ceph-b462a556-226c-11ee-9b33-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-b462a556-226c-11ee-9b33-001a4aab830c@.service. 2023-07-14T17:35:48.655 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:48 smithi088 systemd[1]: Starting Ceph mon.a for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:35:48.868 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T17:35:48.868 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout disabled 2023-07-14T17:35:48.875 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T17:35:48.876 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout inactive 2023-07-14T17:35:48.876 INFO:teuthology.orchestra.run.smithi088.stdout:firewalld.service is not enabled 2023-07-14T17:35:48.876 INFO:teuthology.orchestra.run.smithi088.stdout:Not possible to enable service . firewalld.service is not available 2023-07-14T17:35:48.876 INFO:teuthology.orchestra.run.smithi088.stdout:Waiting for mon to start... 2023-07-14T17:35:48.877 INFO:teuthology.orchestra.run.smithi088.stdout:Waiting for mon... 2023-07-14T17:35:48.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:48 smithi088 podman[126916]: 2023-07-14T17:35:48.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:48 smithi088 bash[126916]: 4ce12687295bc2d7c9085888ff4bb9f6b883e7f5cb9fe0b34fb4328023c20a2e 2023-07-14T17:35:48.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:48 smithi088 systemd[1]: Started Ceph mon.a for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:35:48.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:48 smithi088 ceph-mon[126955]: mkfs b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:35:48.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:48 smithi088 ceph-mon[126955]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T17:35:49.727 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout cluster: 2023-07-14T17:35:49.727 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout id: b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:35:49.727 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-07-14T17:35:49.728 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2023-07-14T17:35:49.728 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout services: 2023-07-14T17:35:49.728 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.508648s) 2023-07-14T17:35:49.728 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-07-14T17:35:49.728 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-07-14T17:35:49.728 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2023-07-14T17:35:49.728 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout data: 2023-07-14T17:35:49.728 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-07-14T17:35:49.729 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-07-14T17:35:49.729 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-07-14T17:35:49.729 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout pgs: 2023-07-14T17:35:49.729 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2023-07-14T17:35:49.729 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.328+0000 7f5ef5def700 1 Processor -- start 2023-07-14T17:35:49.729 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.328+0000 7f5ef5def700 1 -- start start 2023-07-14T17:35:49.729 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.328+0000 7f5ef5def700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ef01045e0 0x7f5ef01049f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:35:49.730 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.328+0000 7f5ef5def700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef0104f30 con 0x7f5ef01045e0 2023-07-14T17:35:49.730 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.329+0000 7f5eef7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ef01045e0 0x7f5ef01049f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:35:49.730 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.329+0000 7f5eef7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ef01045e0 0x7f5ef01049f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40098/0 (socket says 172.21.15.88:40098) 2023-07-14T17:35:49.730 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.329+0000 7f5eef7fe700 1 -- 172.21.15.88:0/294074091 learned_addr learned my addr 172.21.15.88:0/294074091 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:35:49.730 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.329+0000 7f5eef7fe700 1 -- 172.21.15.88:0/294074091 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef0105070 con 0x7f5ef01045e0 2023-07-14T17:35:49.730 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.329+0000 7f5eef7fe700 1 --2- 172.21.15.88:0/294074091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ef01045e0 0x7f5ef01049f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40098a0 tx=0x7f5ee4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58a73454b01e011b server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:35:49.730 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.329+0000 7f5eee7fc700 1 -- 172.21.15.88:0/294074091 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ee4004160 con 0x7f5ef01045e0 2023-07-14T17:35:49.730 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.329+0000 7f5eee7fc700 1 -- 172.21.15.88:0/294074091 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee40042c0 con 0x7f5ef01045e0 2023-07-14T17:35:49.731 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.329+0000 7f5eee7fc700 1 -- 172.21.15.88:0/294074091 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ee40045a0 con 0x7f5ef01045e0 2023-07-14T17:35:49.731 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.330+0000 7f5ef5def700 1 -- 172.21.15.88:0/294074091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ef01045e0 msgr2=0x7f5ef01049f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:35:49.731 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.330+0000 7f5ef5def700 1 --2- 172.21.15.88:0/294074091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ef01045e0 0x7f5ef01049f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5ee40098a0 tx=0x7f5ee4009bb0 comp rx=0 tx=0).stop 2023-07-14T17:35:49.731 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.330+0000 7f5ef5def700 1 -- 172.21.15.88:0/294074091 shutdown_connections 2023-07-14T17:35:49.731 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.330+0000 7f5ef5def700 1 --2- 172.21.15.88:0/294074091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ef01045e0 0x7f5ef01049f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:35:49.731 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.330+0000 7f5ef5def700 1 -- 172.21.15.88:0/294074091 >> 172.21.15.88:0/294074091 conn(0x7f5ef00feef0 msgr2=0x7f5ef0101300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:35:49.731 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.330+0000 7f5ef5def700 1 -- 172.21.15.88:0/294074091 shutdown_connections 2023-07-14T17:35:49.732 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.330+0000 7f5ef5def700 1 -- 172.21.15.88:0/294074091 wait complete. 2023-07-14T17:35:49.732 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.331+0000 7f5ef5def700 1 Processor -- start 2023-07-14T17:35:49.732 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.331+0000 7f5ef5def700 1 -- start start 2023-07-14T17:35:49.732 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.331+0000 7f5ef5def700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ef01045e0 0x7f5ef0197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:35:49.732 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.331+0000 7f5ef5def700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef0197ca0 con 0x7f5ef01045e0 2023-07-14T17:35:49.732 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.331+0000 7f5eef7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ef01045e0 0x7f5ef0197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:35:49.732 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.331+0000 7f5eef7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ef01045e0 0x7f5ef0197760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40100/0 (socket says 172.21.15.88:40100) 2023-07-14T17:35:49.733 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.331+0000 7f5eef7fe700 1 -- 172.21.15.88:0/3796871093 learned_addr learned my addr 172.21.15.88:0/3796871093 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:35:49.733 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.332+0000 7f5eef7fe700 1 -- 172.21.15.88:0/3796871093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee4009550 con 0x7f5ef01045e0 2023-07-14T17:35:49.733 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.332+0000 7f5eef7fe700 1 --2- 172.21.15.88:0/3796871093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ef01045e0 0x7f5ef0197760 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4004000 tx=0x7f5ee4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:35:49.733 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.332+0000 7f5eecff9700 1 -- 172.21.15.88:0/3796871093 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ee4011630 con 0x7f5ef01045e0 2023-07-14T17:35:49.733 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.332+0000 7f5eecff9700 1 -- 172.21.15.88:0/3796871093 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee4011790 con 0x7f5ef01045e0 2023-07-14T17:35:49.733 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.332+0000 7f5eecff9700 1 -- 172.21.15.88:0/3796871093 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ee4011a30 con 0x7f5ef01045e0 2023-07-14T17:35:49.733 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.332+0000 7f5ef5def700 1 -- 172.21.15.88:0/3796871093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef0197ea0 con 0x7f5ef01045e0 2023-07-14T17:35:49.733 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.332+0000 7f5ef5def700 1 -- 172.21.15.88:0/3796871093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef01982c0 con 0x7f5ef01045e0 2023-07-14T17:35:49.734 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.332+0000 7f5eecff9700 1 -- 172.21.15.88:0/3796871093 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5ee4022070 con 0x7f5ef01045e0 2023-07-14T17:35:49.734 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.332+0000 7f5eecff9700 1 -- 172.21.15.88:0/3796871093 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5ee401c450 con 0x7f5ef01045e0 2023-07-14T17:35:49.734 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.333+0000 7f5ef5def700 1 -- 172.21.15.88:0/3796871093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef01054c0 con 0x7f5ef01045e0 2023-07-14T17:35:49.734 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.335+0000 7f5eecff9700 1 -- 172.21.15.88:0/3796871093 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f5ee401b450 con 0x7f5ef01045e0 2023-07-14T17:35:49.734 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.404+0000 7f5ef5def700 1 -- 172.21.15.88:0/3796871093 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f5ef0190cc0 con 0x7f5ef01045e0 2023-07-14T17:35:49.734 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.405+0000 7f5eecff9700 1 -- 172.21.15.88:0/3796871093 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f5ee401b5f0 con 0x7f5ef01045e0 2023-07-14T17:35:49.734 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.405+0000 7f5ef5def700 1 -- 172.21.15.88:0/3796871093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ef01045e0 msgr2=0x7f5ef0197760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:35:49.735 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.405+0000 7f5ef5def700 1 --2- 172.21.15.88:0/3796871093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ef01045e0 0x7f5ef0197760 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4004000 tx=0x7f5ee4004750 comp rx=0 tx=0).stop 2023-07-14T17:35:49.735 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.406+0000 7f5ef5def700 1 -- 172.21.15.88:0/3796871093 shutdown_connections 2023-07-14T17:35:49.735 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.406+0000 7f5ef5def700 1 --2- 172.21.15.88:0/3796871093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ef01045e0 0x7f5ef0197760 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:35:49.735 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.406+0000 7f5ef5def700 1 -- 172.21.15.88:0/3796871093 >> 172.21.15.88:0/3796871093 conn(0x7f5ef00feef0 msgr2=0x7f5ef018bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:35:49.735 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.406+0000 7f5ef5def700 1 -- 172.21.15.88:0/3796871093 shutdown_connections 2023-07-14T17:35:49.735 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:49.406+0000 7f5ef5def700 1 -- 172.21.15.88:0/3796871093 wait complete. 2023-07-14T17:35:49.735 INFO:teuthology.orchestra.run.smithi088.stdout:mon is available 2023-07-14T17:35:49.736 INFO:teuthology.orchestra.run.smithi088.stdout:Assimilating anything we can from ceph.conf... 2023-07-14T17:35:49.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:49 smithi088 ceph-mon[126955]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T17:35:49.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:49 smithi088 ceph-mon[126955]: monmap e1: 1 mons at {a=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0]} removed_ranks: {} 2023-07-14T17:35:49.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:49 smithi088 ceph-mon[126955]: fsmap 2023-07-14T17:35:49.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:49 smithi088 ceph-mon[126955]: osdmap e1: 0 total, 0 up, 0 in 2023-07-14T17:35:49.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:49 smithi088 ceph-mon[126955]: mgrmap e1: no daemons active 2023-07-14T17:35:49.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:49 smithi088 ceph-mon[126955]: from='client.? 172.21.15.88:0/3796871093' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-14T17:35:50.670 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2023-07-14T17:35:50.671 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout [global] 2023-07-14T17:35:50.671 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout fsid = b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:35:50.671 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.88:3300,v1:172.21.15.88:6789] 2023-07-14T17:35:50.671 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-14T17:35:50.671 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-14T17:35:50.671 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-14T17:35:50.671 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-14T17:35:50.672 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2023-07-14T17:35:50.672 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-14T17:35:50.672 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-14T17:35:50.672 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2023-07-14T17:35:50.672 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout [osd] 2023-07-14T17:35:50.672 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-14T17:35:50.672 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-14T17:35:50.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-14T17:35:50.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-14T17:35:50.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-14T17:35:50.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.213+0000 7fa6d5776700 1 Processor -- start 2023-07-14T17:35:50.673 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.213+0000 7fa6d5776700 1 -- start start 2023-07-14T17:35:50.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.213+0000 7fa6d5776700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d01034d0 0x7fa6d01038e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:35:50.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.213+0000 7fa6d5776700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d0103e20 con 0x7fa6d01034d0 2023-07-14T17:35:50.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.214+0000 7fa6ceffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d01034d0 0x7fa6d01038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:35:50.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.214+0000 7fa6ceffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d01034d0 0x7fa6d01038e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40102/0 (socket says 172.21.15.88:40102) 2023-07-14T17:35:50.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.214+0000 7fa6ceffd700 1 -- 172.21.15.88:0/1243451758 learned_addr learned my addr 172.21.15.88:0/1243451758 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:35:50.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.214+0000 7fa6ceffd700 1 -- 172.21.15.88:0/1243451758 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d0103f60 con 0x7fa6d01034d0 2023-07-14T17:35:50.674 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.214+0000 7fa6ceffd700 1 --2- 172.21.15.88:0/1243451758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d01034d0 0x7fa6d01038e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40098a0 tx=0x7fa6c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7447fbc072df01d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:35:50.675 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.214+0000 7fa6cdffb700 1 -- 172.21.15.88:0/1243451758 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6c4004160 con 0x7fa6d01034d0 2023-07-14T17:35:50.675 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.214+0000 7fa6cdffb700 1 -- 172.21.15.88:0/1243451758 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c40042c0 con 0x7fa6d01034d0 2023-07-14T17:35:50.675 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.214+0000 7fa6cdffb700 1 -- 172.21.15.88:0/1243451758 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6c40045a0 con 0x7fa6d01034d0 2023-07-14T17:35:50.675 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.215+0000 7fa6d5776700 1 -- 172.21.15.88:0/1243451758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d01034d0 msgr2=0x7fa6d01038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:35:50.675 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.215+0000 7fa6d5776700 1 --2- 172.21.15.88:0/1243451758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d01034d0 0x7fa6d01038e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40098a0 tx=0x7fa6c4009bb0 comp rx=0 tx=0).stop 2023-07-14T17:35:50.675 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.215+0000 7fa6d5776700 1 -- 172.21.15.88:0/1243451758 shutdown_connections 2023-07-14T17:35:50.675 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.215+0000 7fa6d5776700 1 --2- 172.21.15.88:0/1243451758 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d01034d0 0x7fa6d01038e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:35:50.676 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.215+0000 7fa6d5776700 1 -- 172.21.15.88:0/1243451758 >> 172.21.15.88:0/1243451758 conn(0x7fa6d00feef0 msgr2=0x7fa6d0101300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:35:50.676 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.215+0000 7fa6d5776700 1 -- 172.21.15.88:0/1243451758 shutdown_connections 2023-07-14T17:35:50.676 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.215+0000 7fa6d5776700 1 -- 172.21.15.88:0/1243451758 wait complete. 2023-07-14T17:35:50.676 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.215+0000 7fa6d5776700 1 Processor -- start 2023-07-14T17:35:50.676 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.215+0000 7fa6d5776700 1 -- start start 2023-07-14T17:35:50.676 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.216+0000 7fa6d5776700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d01034d0 0x7fa6d0198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:35:50.676 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.216+0000 7fa6d5776700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d0198dc0 con 0x7fa6d01034d0 2023-07-14T17:35:50.677 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.216+0000 7fa6ceffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d01034d0 0x7fa6d0198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:35:50.677 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.216+0000 7fa6ceffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d01034d0 0x7fa6d0198880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40104/0 (socket says 172.21.15.88:40104) 2023-07-14T17:35:50.677 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.216+0000 7fa6ceffd700 1 -- 172.21.15.88:0/890564267 learned_addr learned my addr 172.21.15.88:0/890564267 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:35:50.677 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.216+0000 7fa6ceffd700 1 -- 172.21.15.88:0/890564267 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c4009550 con 0x7fa6d01034d0 2023-07-14T17:35:50.677 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.216+0000 7fa6ceffd700 1 --2- 172.21.15.88:0/890564267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d01034d0 0x7fa6d0198880 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4004000 tx=0x7fa6c4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:35:50.677 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.216+0000 7fa6b7fff700 1 -- 172.21.15.88:0/890564267 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6c4011630 con 0x7fa6d01034d0 2023-07-14T17:35:50.677 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.216+0000 7fa6b7fff700 1 -- 172.21.15.88:0/890564267 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c4011790 con 0x7fa6d01034d0 2023-07-14T17:35:50.678 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.216+0000 7fa6b7fff700 1 -- 172.21.15.88:0/890564267 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6c4011a30 con 0x7fa6d01034d0 2023-07-14T17:35:50.678 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.216+0000 7fa6d5776700 1 -- 172.21.15.88:0/890564267 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d0198fc0 con 0x7fa6d01034d0 2023-07-14T17:35:50.678 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.216+0000 7fa6d5776700 1 -- 172.21.15.88:0/890564267 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d01993e0 con 0x7fa6d01034d0 2023-07-14T17:35:50.678 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.217+0000 7fa6b7fff700 1 -- 172.21.15.88:0/890564267 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa6c4022070 con 0x7fa6d01034d0 2023-07-14T17:35:50.678 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.217+0000 7fa6b7fff700 1 -- 172.21.15.88:0/890564267 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa6c401c450 con 0x7fa6d01034d0 2023-07-14T17:35:50.678 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.217+0000 7fa6d5776700 1 -- 172.21.15.88:0/890564267 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d01065c0 con 0x7fa6d01034d0 2023-07-14T17:35:50.678 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.220+0000 7fa6b7fff700 1 -- 172.21.15.88:0/890564267 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fa6c401b450 con 0x7fa6d01034d0 2023-07-14T17:35:50.678 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.286+0000 7fa6d5776700 1 -- 172.21.15.88:0/890564267 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa6d0106780 con 0x7fa6d01034d0 2023-07-14T17:35:50.679 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.290+0000 7fa6b7fff700 1 -- 172.21.15.88:0/890564267 <== mon.0 v2:172.21.15.88:3300/0 7 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fa6c401b5f0 con 0x7fa6d01034d0 2023-07-14T17:35:50.679 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.290+0000 7fa6b7fff700 1 -- 172.21.15.88:0/890564267 <== mon.0 v2:172.21.15.88:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fa6c4024560 con 0x7fa6d01034d0 2023-07-14T17:35:50.679 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.291+0000 7fa6d5776700 1 -- 172.21.15.88:0/890564267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d01034d0 msgr2=0x7fa6d0198880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:35:50.679 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.291+0000 7fa6d5776700 1 --2- 172.21.15.88:0/890564267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d01034d0 0x7fa6d0198880 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4004000 tx=0x7fa6c4004750 comp rx=0 tx=0).stop 2023-07-14T17:35:50.679 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.291+0000 7fa6d5776700 1 -- 172.21.15.88:0/890564267 shutdown_connections 2023-07-14T17:35:50.679 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.291+0000 7fa6d5776700 1 --2- 172.21.15.88:0/890564267 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6d01034d0 0x7fa6d0198880 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:35:50.679 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.291+0000 7fa6d5776700 1 -- 172.21.15.88:0/890564267 >> 172.21.15.88:0/890564267 conn(0x7fa6d00feef0 msgr2=0x7fa6d018ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:35:50.680 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.291+0000 7fa6d5776700 1 -- 172.21.15.88:0/890564267 shutdown_connections 2023-07-14T17:35:50.680 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:50.291+0000 7fa6d5776700 1 -- 172.21.15.88:0/890564267 wait complete. 2023-07-14T17:35:50.680 INFO:teuthology.orchestra.run.smithi088.stdout:Generating new minimal ceph.conf... 2023-07-14T17:35:51.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:50 smithi088 ceph-mon[126955]: from='client.? 172.21.15.88:0/890564267' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-14T17:35:51.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:50 smithi088 ceph-mon[126955]: from='client.? 172.21.15.88:0/890564267' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-14T17:35:51.615 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.162+0000 7f3e068a8700 1 Processor -- start 2023-07-14T17:35:51.615 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.162+0000 7f3e068a8700 1 -- start start 2023-07-14T17:35:51.615 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.163+0000 7f3e068a8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e00105d60 0x7f3e00108140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:35:51.615 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.163+0000 7f3e068a8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e00079cf0 con 0x7f3e00105d60 2023-07-14T17:35:51.615 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.163+0000 7f3dfffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e00105d60 0x7f3e00108140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:35:51.616 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.163+0000 7f3dfffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e00105d60 0x7f3e00108140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40106/0 (socket says 172.21.15.88:40106) 2023-07-14T17:35:51.616 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.163+0000 7f3dfffff700 1 -- 172.21.15.88:0/1852461307 learned_addr learned my addr 172.21.15.88:0/1852461307 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:35:51.616 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.163+0000 7f3dfffff700 1 -- 172.21.15.88:0/1852461307 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e00079e30 con 0x7f3e00105d60 2023-07-14T17:35:51.616 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.163+0000 7f3dfffff700 1 --2- 172.21.15.88:0/1852461307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e00105d60 0x7f3e00108140 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3df00098a0 tx=0x7f3df0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=153c0746db13c6f5 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:35:51.616 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.163+0000 7f3dfeffd700 1 -- 172.21.15.88:0/1852461307 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3df0004160 con 0x7f3e00105d60 2023-07-14T17:35:51.616 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.163+0000 7f3dfeffd700 1 -- 172.21.15.88:0/1852461307 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f3df00042c0 con 0x7f3e00105d60 2023-07-14T17:35:51.617 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.164+0000 7f3dfeffd700 1 -- 172.21.15.88:0/1852461307 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3df0011940 con 0x7f3e00105d60 2023-07-14T17:35:51.617 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.164+0000 7f3e068a8700 1 -- 172.21.15.88:0/1852461307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e00105d60 msgr2=0x7f3e00108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:35:51.617 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.164+0000 7f3e068a8700 1 --2- 172.21.15.88:0/1852461307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e00105d60 0x7f3e00108140 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3df00098a0 tx=0x7f3df0009bb0 comp rx=0 tx=0).stop 2023-07-14T17:35:51.617 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.164+0000 7f3e068a8700 1 -- 172.21.15.88:0/1852461307 shutdown_connections 2023-07-14T17:35:51.617 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.164+0000 7f3e068a8700 1 --2- 172.21.15.88:0/1852461307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e00105d60 0x7f3e00108140 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:35:51.617 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.164+0000 7f3e068a8700 1 -- 172.21.15.88:0/1852461307 >> 172.21.15.88:0/1852461307 conn(0x7f3e000ff7a0 msgr2=0x7f3e00101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:35:51.617 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.164+0000 7f3e068a8700 1 -- 172.21.15.88:0/1852461307 shutdown_connections 2023-07-14T17:35:51.617 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.164+0000 7f3e068a8700 1 -- 172.21.15.88:0/1852461307 wait complete. 2023-07-14T17:35:51.618 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.165+0000 7f3e068a8700 1 Processor -- start 2023-07-14T17:35:51.618 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.165+0000 7f3e068a8700 1 -- start start 2023-07-14T17:35:51.618 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.165+0000 7f3e068a8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e00105d60 0x7f3e00198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:35:51.618 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.165+0000 7f3e068a8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e00199490 con 0x7f3e00105d60 2023-07-14T17:35:51.618 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.165+0000 7f3dfffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e00105d60 0x7f3e00198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:35:51.618 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.165+0000 7f3dfffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e00105d60 0x7f3e00198f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40108/0 (socket says 172.21.15.88:40108) 2023-07-14T17:35:51.618 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.165+0000 7f3dfffff700 1 -- 172.21.15.88:0/2471953074 learned_addr learned my addr 172.21.15.88:0/2471953074 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:35:51.619 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.165+0000 7f3dfffff700 1 -- 172.21.15.88:0/2471953074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df0009550 con 0x7f3e00105d60 2023-07-14T17:35:51.619 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.166+0000 7f3dfffff700 1 --2- 172.21.15.88:0/2471953074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e00105d60 0x7f3e00198f50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3df0006af0 tx=0x7f3df0011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:35:51.619 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.166+0000 7f3dfd7fa700 1 -- 172.21.15.88:0/2471953074 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3df0003bf0 con 0x7f3e00105d60 2023-07-14T17:35:51.619 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.166+0000 7f3dfd7fa700 1 -- 172.21.15.88:0/2471953074 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f3df001b8b0 con 0x7f3e00105d60 2023-07-14T17:35:51.619 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.166+0000 7f3e068a8700 1 -- 172.21.15.88:0/2471953074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e00199690 con 0x7f3e00105d60 2023-07-14T17:35:51.619 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.166+0000 7f3dfd7fa700 1 -- 172.21.15.88:0/2471953074 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3df00198b0 con 0x7f3e00105d60 2023-07-14T17:35:51.619 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.166+0000 7f3e068a8700 1 -- 172.21.15.88:0/2471953074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e00199ab0 con 0x7f3e00105d60 2023-07-14T17:35:51.619 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.166+0000 7f3dfd7fa700 1 -- 172.21.15.88:0/2471953074 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3df00233f0 con 0x7f3e00105d60 2023-07-14T17:35:51.620 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.166+0000 7f3dfd7fa700 1 -- 172.21.15.88:0/2471953074 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3df001a710 con 0x7f3e00105d60 2023-07-14T17:35:51.620 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.167+0000 7f3e068a8700 1 -- 172.21.15.88:0/2471953074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e00109290 con 0x7f3e00105d60 2023-07-14T17:35:51.620 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.169+0000 7f3dfd7fa700 1 -- 172.21.15.88:0/2471953074 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f3df0028060 con 0x7f3e00105d60 2023-07-14T17:35:51.620 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.236+0000 7f3e068a8700 1 -- 172.21.15.88:0/2471953074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f3e00109480 con 0x7f3e00105d60 2023-07-14T17:35:51.620 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.237+0000 7f3dfd7fa700 1 -- 172.21.15.88:0/2471953074 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f3df001bd70 con 0x7f3e00105d60 2023-07-14T17:35:51.620 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.238+0000 7f3e068a8700 1 -- 172.21.15.88:0/2471953074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e00105d60 msgr2=0x7f3e00198f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:35:51.620 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.238+0000 7f3e068a8700 1 --2- 172.21.15.88:0/2471953074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e00105d60 0x7f3e00198f50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3df0006af0 tx=0x7f3df0011a30 comp rx=0 tx=0).stop 2023-07-14T17:35:51.621 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.238+0000 7f3e068a8700 1 -- 172.21.15.88:0/2471953074 shutdown_connections 2023-07-14T17:35:51.621 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.238+0000 7f3e068a8700 1 --2- 172.21.15.88:0/2471953074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e00105d60 0x7f3e00198f50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:35:51.621 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.238+0000 7f3e068a8700 1 -- 172.21.15.88:0/2471953074 >> 172.21.15.88:0/2471953074 conn(0x7f3e000ff7a0 msgr2=0x7f3e00100560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:35:51.621 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.238+0000 7f3e068a8700 1 -- 172.21.15.88:0/2471953074 shutdown_connections 2023-07-14T17:35:51.621 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:51.238+0000 7f3e068a8700 1 -- 172.21.15.88:0/2471953074 wait complete. 2023-07-14T17:35:51.621 INFO:teuthology.orchestra.run.smithi088.stdout:Restarting the monitor... 2023-07-14T17:35:51.892 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:51 smithi088 systemd[1]: Stopping Ceph mon.a for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:35:52.144 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:51 smithi088 ceph-mon[126955]: from='client.? 172.21.15.88:0/2471953074' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:35:52.144 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:51 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mon-a[126931]: 2023-07-14T17:35:51.968+0000 7f8dd3e1a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:35:52.144 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:51 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mon-a[126931]: 2023-07-14T17:35:51.968+0000 7f8dd3e1a700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-14T17:35:52.401 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:52 smithi088 bash[127302]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-mon-a 2023-07-14T17:35:52.697 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:52 smithi088 systemd[1]: ceph-b462a556-226c-11ee-9b33-001a4aab830c@mon.a.service: Succeeded. 2023-07-14T17:35:52.697 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:52 smithi088 systemd[1]: Stopped Ceph mon.a for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:35:52.697 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:52 smithi088 systemd[1]: Starting Ceph mon.a for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:35:54.447 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:54 smithi088 podman[127425]: 2023-07-14T17:35:55.682 INFO:teuthology.orchestra.run.smithi088.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-07-14T17:35:55.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T17:35:55.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T17:35:55.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: pidfile_write: ignore empty --pid-file 2023-07-14T17:35:55.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: load: jerasure load: lrc 2023-07-14T17:35:55.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: RocksDB version: 7.9.2 2023-07-14T17:35:55.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Git sha 0 2023-07-14T17:35:55.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T17:35:55.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: DB SUMMARY 2023-07-14T17:35:55.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: DB Session ID: KUGML52N55VHMIZ8XO84 2023-07-14T17:35:55.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: CURRENT file: CURRENT 2023-07-14T17:35:55.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T17:35:55.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-07-14T17:35:55.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-07-14T17:35:55.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 97706 ; 2023-07-14T17:35:55.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.error_if_exists: 0 2023-07-14T17:35:55.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.create_if_missing: 0 2023-07-14T17:35:55.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.paranoid_checks: 1 2023-07-14T17:35:55.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T17:35:55.950 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T17:35:55.950 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T17:35:55.950 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.env: 0x55b1ac4d1c60 2023-07-14T17:35:55.950 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T17:35:55.950 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.info_log: 0x55b1ad3dd2a0 2023-07-14T17:35:55.950 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T17:35:55.950 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.statistics: (nil) 2023-07-14T17:35:55.951 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.use_fsync: 0 2023-07-14T17:35:55.951 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_log_file_size: 0 2023-07-14T17:35:55.951 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T17:35:55.951 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T17:35:55.951 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T17:35:55.951 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T17:35:55.951 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.allow_fallocate: 1 2023-07-14T17:35:55.952 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T17:35:55.952 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T17:35:55.952 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.use_direct_reads: 0 2023-07-14T17:35:55.952 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T17:35:55.953 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T17:35:55.953 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.db_log_dir: 2023-07-14T17:35:55.953 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.wal_dir: 2023-07-14T17:35:55.953 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T17:35:55.953 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T17:35:55.953 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T17:35:55.953 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T17:35:55.953 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T17:35:55.954 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T17:35:55.954 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T17:35:55.954 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T17:35:55.954 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.write_buffer_manager: 0x55b1ad2fe3c0 2023-07-14T17:35:55.954 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T17:35:55.954 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T17:35:55.954 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T17:35:55.955 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T17:35:55.955 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T17:35:55.955 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T17:35:55.955 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T17:35:55.955 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T17:35:55.955 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.unordered_write: 0 2023-07-14T17:35:55.955 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T17:35:55.955 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T17:35:55.956 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T17:35:55.956 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T17:35:55.956 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.row_cache: None 2023-07-14T17:35:55.956 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.wal_filter: None 2023-07-14T17:35:55.956 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T17:35:55.956 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T17:35:55.956 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.two_write_queues: 0 2023-07-14T17:35:55.957 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T17:35:55.957 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.wal_compression: 0 2023-07-14T17:35:55.957 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.atomic_flush: 0 2023-07-14T17:35:55.957 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T17:35:55.957 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T17:35:55.957 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T17:35:55.957 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.log_readahead_size: 0 2023-07-14T17:35:55.958 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T17:35:55.958 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T17:35:55.958 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T17:35:55.958 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T17:35:55.958 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T17:35:55.958 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T17:35:55.959 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T17:35:55.960 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_background_jobs: 2 2023-07-14T17:35:55.960 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_background_compactions: -1 2023-07-14T17:35:55.960 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_subcompactions: 1 2023-07-14T17:35:55.960 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T17:35:55.960 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T17:35:55.960 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T17:35:55.961 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T17:35:55.961 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T17:35:55.961 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T17:35:55.961 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T17:35:55.961 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T17:35:55.961 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_open_files: -1 2023-07-14T17:35:55.961 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T17:35:55.961 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T17:35:55.962 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T17:35:55.962 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T17:35:55.962 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_background_flushes: -1 2023-07-14T17:35:55.962 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Compression algorithms supported: 2023-07-14T17:35:55.962 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T17:35:55.962 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: kZSTD supported: 0 2023-07-14T17:35:55.962 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: kXpressCompression supported: 0 2023-07-14T17:35:55.962 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T17:35:55.963 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: kZlibCompression supported: 1 2023-07-14T17:35:55.963 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: kSnappyCompression supported: 1 2023-07-14T17:35:55.963 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: kLZ4Compression supported: 1 2023-07-14T17:35:55.963 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: kBZip2Compression supported: 0 2023-07-14T17:35:55.963 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T17:35:55.963 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T17:35:55.963 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-07-14T17:35:55.964 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T17:35:55.964 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T17:35:55.964 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.merge_operator: 2023-07-14T17:35:55.964 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compaction_filter: None 2023-07-14T17:35:55.964 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T17:35:55.964 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T17:35:55.964 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T17:35:55.964 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T17:35:55.965 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b1ad3dd3a0) 2023-07-14T17:35:55.965 INFO:journalctl@ceph.mon.a.smithi088.stdout: cache_index_and_filter_blocks: 1 2023-07-14T17:35:55.965 INFO:journalctl@ceph.mon.a.smithi088.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T17:35:55.965 INFO:journalctl@ceph.mon.a.smithi088.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T17:35:55.965 INFO:journalctl@ceph.mon.a.smithi088.stdout: pin_top_level_index_and_filter: 1 2023-07-14T17:35:55.965 INFO:journalctl@ceph.mon.a.smithi088.stdout: index_type: 0 2023-07-14T17:35:55.965 INFO:journalctl@ceph.mon.a.smithi088.stdout: data_block_index_type: 0 2023-07-14T17:35:55.965 INFO:journalctl@ceph.mon.a.smithi088.stdout: index_shortening: 1 2023-07-14T17:35:55.966 INFO:journalctl@ceph.mon.a.smithi088.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T17:35:55.966 INFO:journalctl@ceph.mon.a.smithi088.stdout: checksum: 4 2023-07-14T17:35:55.966 INFO:journalctl@ceph.mon.a.smithi088.stdout: no_block_cache: 0 2023-07-14T17:35:55.966 INFO:journalctl@ceph.mon.a.smithi088.stdout: block_cache: 0x55b1ad39d1f0 2023-07-14T17:35:55.966 INFO:journalctl@ceph.mon.a.smithi088.stdout: block_cache_name: BinnedLRUCache 2023-07-14T17:35:55.966 INFO:journalctl@ceph.mon.a.smithi088.stdout: block_cache_options: 2023-07-14T17:35:55.966 INFO:journalctl@ceph.mon.a.smithi088.stdout: capacity : 536870912 2023-07-14T17:35:55.967 INFO:journalctl@ceph.mon.a.smithi088.stdout: num_shard_bits : 4 2023-07-14T17:35:55.967 INFO:journalctl@ceph.mon.a.smithi088.stdout: strict_capacity_limit : 0 2023-07-14T17:35:55.967 INFO:journalctl@ceph.mon.a.smithi088.stdout: high_pri_pool_ratio: 0.000 2023-07-14T17:35:55.967 INFO:journalctl@ceph.mon.a.smithi088.stdout: block_cache_compressed: (nil) 2023-07-14T17:35:55.967 INFO:journalctl@ceph.mon.a.smithi088.stdout: persistent_cache: (nil) 2023-07-14T17:35:55.967 INFO:journalctl@ceph.mon.a.smithi088.stdout: block_size: 4096 2023-07-14T17:35:55.967 INFO:journalctl@ceph.mon.a.smithi088.stdout: block_size_deviation: 10 2023-07-14T17:35:55.967 INFO:journalctl@ceph.mon.a.smithi088.stdout: block_restart_interval: 16 2023-07-14T17:35:55.968 INFO:journalctl@ceph.mon.a.smithi088.stdout: index_block_restart_interval: 1 2023-07-14T17:35:55.968 INFO:journalctl@ceph.mon.a.smithi088.stdout: metadata_block_size: 4096 2023-07-14T17:35:55.968 INFO:journalctl@ceph.mon.a.smithi088.stdout: partition_filters: 0 2023-07-14T17:35:55.968 INFO:journalctl@ceph.mon.a.smithi088.stdout: use_delta_encoding: 1 2023-07-14T17:35:55.968 INFO:journalctl@ceph.mon.a.smithi088.stdout: filter_policy: bloomfilter 2023-07-14T17:35:55.968 INFO:journalctl@ceph.mon.a.smithi088.stdout: whole_key_filtering: 1 2023-07-14T17:35:55.968 INFO:journalctl@ceph.mon.a.smithi088.stdout: verify_compression: 0 2023-07-14T17:35:55.968 INFO:journalctl@ceph.mon.a.smithi088.stdout: read_amp_bytes_per_bit: 0 2023-07-14T17:35:55.969 INFO:journalctl@ceph.mon.a.smithi088.stdout: format_version: 5 2023-07-14T17:35:55.969 INFO:journalctl@ceph.mon.a.smithi088.stdout: enable_index_compression: 1 2023-07-14T17:35:55.969 INFO:journalctl@ceph.mon.a.smithi088.stdout: block_align: 0 2023-07-14T17:35:55.969 INFO:journalctl@ceph.mon.a.smithi088.stdout: max_auto_readahead_size: 262144 2023-07-14T17:35:55.969 INFO:journalctl@ceph.mon.a.smithi088.stdout: prepopulate_block_cache: 0 2023-07-14T17:35:55.969 INFO:journalctl@ceph.mon.a.smithi088.stdout: initial_auto_readahead_size: 8192 2023-07-14T17:35:55.969 INFO:journalctl@ceph.mon.a.smithi088.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T17:35:55.970 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T17:35:55.970 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T17:35:55.970 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compression: NoCompression 2023-07-14T17:35:55.970 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T17:35:55.970 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T17:35:55.970 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T17:35:55.970 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.num_levels: 7 2023-07-14T17:35:55.970 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T17:35:55.971 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T17:35:55.971 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T17:35:55.971 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T17:35:55.971 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T17:35:55.971 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T17:35:55.971 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T17:35:55.971 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T17:35:55.971 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T17:35:55.972 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T17:35:55.972 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T17:35:55.972 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T17:35:55.972 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T17:35:55.972 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T17:35:55.972 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T17:35:55.972 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T17:35:55.973 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T17:35:55.973 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T17:35:55.973 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T17:35:55.973 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T17:35:55.973 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T17:35:55.973 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T17:35:55.973 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T17:35:55.973 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T17:35:55.974 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T17:35:55.974 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T17:35:55.974 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T17:35:55.974 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T17:35:55.974 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T17:35:55.974 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T17:35:55.974 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T17:35:55.974 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T17:35:55.975 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T17:35:55.975 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T17:35:55.975 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T17:35:55.975 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T17:35:55.975 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T17:35:55.975 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T17:35:55.975 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T17:35:55.975 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T17:35:55.976 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T17:35:55.976 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T17:35:55.976 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T17:35:55.976 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T17:35:55.976 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T17:35:55.976 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T17:35:55.976 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T17:35:55.977 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T17:35:55.977 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T17:35:55.977 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T17:35:55.977 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T17:35:55.977 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T17:35:55.977 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T17:35:55.977 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.table_properties_collectors: 2023-07-14T17:35:55.978 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.inplace_update_support: 0 2023-07-14T17:35:55.978 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T17:35:55.978 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T17:35:55.978 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T17:35:55.978 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T17:35:55.978 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.bloom_locality: 0 2023-07-14T17:35:55.978 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.max_successive_merges: 0 2023-07-14T17:35:55.978 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T17:35:55.979 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T17:35:55.979 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T17:35:55.979 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T17:35:55.979 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.ttl: 2592000 2023-07-14T17:35:55.979 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T17:35:55.979 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T17:35:55.979 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T17:35:55.980 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.enable_blob_files: false 2023-07-14T17:35:55.980 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.min_blob_size: 0 2023-07-14T17:35:55.980 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T17:35:55.980 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T17:35:55.980 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T17:35:55.980 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T17:35:55.980 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T17:35:55.980 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T17:35:55.981 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T17:35:55.981 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T17:35:55.981 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-07-14T17:35:55.981 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-07-14T17:35:55.981 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 234d48f6-4418-4188-910e-1677ead2c7a6 2023-07-14T17:35:55.981 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689356155594581, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-07-14T17:35:55.981 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-07-14T17:35:55.982 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689356155595634, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93038, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 310, "table_properties": {"data_size": 91060, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13396, "raw_average_key_size": 49, "raw_value_size": 83748, "raw_average_value_size": 311, "num_data_blocks": 11, "num_entries": 269, "num_filter_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689356155, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "234d48f6-4418-4188-910e-1677ead2c7a6", "db_session_id": "KUGML52N55VHMIZ8XO84", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-07-14T17:35:55.982 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689356155595713, "job": 1, "event": "recovery_finished"} 2023-07-14T17:35:55.982 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-07-14T17:35:55.982 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T17:35:55.982 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b1ad436000 2023-07-14T17:35:55.982 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: DB pointer 0x55b1ad422000 2023-07-14T17:35:55.982 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T17:35:55.983 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T17:35:55.983 INFO:journalctl@ceph.mon.a.smithi088.stdout: ** DB Stats ** 2023-07-14T17:35:55.983 INFO:journalctl@ceph.mon.a.smithi088.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T17:35:55.983 INFO:journalctl@ceph.mon.a.smithi088.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T17:35:55.983 INFO:journalctl@ceph.mon.a.smithi088.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T17:35:55.983 INFO:journalctl@ceph.mon.a.smithi088.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T17:35:55.983 INFO:journalctl@ceph.mon.a.smithi088.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T17:35:55.983 INFO:journalctl@ceph.mon.a.smithi088.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T17:35:55.984 INFO:journalctl@ceph.mon.a.smithi088.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T17:35:55.984 INFO:journalctl@ceph.mon.a.smithi088.stdout: 2023-07-14T17:35:55.984 INFO:journalctl@ceph.mon.a.smithi088.stdout: ** Compaction Stats [default] ** 2023-07-14T17:35:55.984 INFO:journalctl@ceph.mon.a.smithi088.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T17:35:55.984 INFO:journalctl@ceph.mon.a.smithi088.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T17:35:55.984 INFO:journalctl@ceph.mon.a.smithi088.stdout: L0 2/0 92.69 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T17:35:55.984 INFO:journalctl@ceph.mon.a.smithi088.stdout: Sum 2/0 92.69 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T17:35:55.985 INFO:journalctl@ceph.mon.a.smithi088.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 128.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T17:35:55.985 INFO:journalctl@ceph.mon.a.smithi088.stdout: 2023-07-14T17:35:55.985 INFO:journalctl@ceph.mon.a.smithi088.stdout: ** Compaction Stats [default] ** 2023-07-14T17:35:55.985 INFO:journalctl@ceph.mon.a.smithi088.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T17:35:55.985 INFO:journalctl@ceph.mon.a.smithi088.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T17:35:55.985 INFO:journalctl@ceph.mon.a.smithi088.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 128.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T17:35:55.985 INFO:journalctl@ceph.mon.a.smithi088.stdout: 2023-07-14T17:35:55.985 INFO:journalctl@ceph.mon.a.smithi088.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T17:35:55.986 INFO:journalctl@ceph.mon.a.smithi088.stdout: 2023-07-14T17:35:55.986 INFO:journalctl@ceph.mon.a.smithi088.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T17:35:55.986 INFO:journalctl@ceph.mon.a.smithi088.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T17:35:55.986 INFO:journalctl@ceph.mon.a.smithi088.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T17:35:55.986 INFO:journalctl@ceph.mon.a.smithi088.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T17:35:55.986 INFO:journalctl@ceph.mon.a.smithi088.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T17:35:55.986 INFO:journalctl@ceph.mon.a.smithi088.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T17:35:55.986 INFO:journalctl@ceph.mon.a.smithi088.stdout: Cumulative compaction: 0.00 GB write, 15.83 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T17:35:55.987 INFO:journalctl@ceph.mon.a.smithi088.stdout: Interval compaction: 0.00 GB write, 15.83 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T17:35:55.987 INFO:journalctl@ceph.mon.a.smithi088.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T17:35:55.987 INFO:journalctl@ceph.mon.a.smithi088.stdout: Block cache BinnedLRUCache@0x55b1ad39d1f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2023-07-14T17:35:55.987 INFO:journalctl@ceph.mon.a.smithi088.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-07-14T17:35:55.987 INFO:journalctl@ceph.mon.a.smithi088.stdout: 2023-07-14T17:35:55.987 INFO:journalctl@ceph.mon.a.smithi088.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T17:35:55.987 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: starting mon.a rank 0 at public addrs [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] at bind addrs [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:35:55.988 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: mon.a@-1(???) e1 preinit fsid b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:35:55.988 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: mon.a@-1(???).mds e1 new map 2023-07-14T17:35:55.988 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: mon.a@-1(???).mds e1 print_map 2023-07-14T17:35:55.988 INFO:journalctl@ceph.mon.a.smithi088.stdout: e1 2023-07-14T17:35:55.988 INFO:journalctl@ceph.mon.a.smithi088.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T17:35:55.988 INFO:journalctl@ceph.mon.a.smithi088.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T17:35:55.988 INFO:journalctl@ceph.mon.a.smithi088.stdout: legacy client fscid: -1 2023-07-14T17:35:55.988 INFO:journalctl@ceph.mon.a.smithi088.stdout: 2023-07-14T17:35:55.989 INFO:journalctl@ceph.mon.a.smithi088.stdout: No filesystems configured 2023-07-14T17:35:55.989 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T17:35:55.989 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T17:35:55.989 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T17:35:55.989 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T17:35:55.989 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-14T17:35:55.989 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: expand_channel_meta expand map: {default=false} 2023-07-14T17:35:55.990 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: expand_channel_meta from 'false' to 'false' 2023-07-14T17:35:55.990 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: expand_channel_meta expanded map: {default=false} 2023-07-14T17:35:55.990 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: expand_channel_meta expand map: {default=info} 2023-07-14T17:35:55.990 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: expand_channel_meta from 'info' to 'info' 2023-07-14T17:35:55.990 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: expand_channel_meta expanded map: {default=info} 2023-07-14T17:35:55.990 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: expand_channel_meta expand map: {default=daemon} 2023-07-14T17:35:55.990 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-14T17:35:55.990 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: expand_channel_meta expanded map: {default=daemon} 2023-07-14T17:35:55.991 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: expand_channel_meta expand map: {default=debug} 2023-07-14T17:35:55.991 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: expand_channel_meta from 'debug' to 'debug' 2023-07-14T17:35:55.991 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: expand_channel_meta expanded map: {default=debug} 2023-07-14T17:35:55.991 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 bash[127425]: 308edc0019a3d790e78eecab46c0ef5c05b4982a2dda53052c486c0781b32e3d 2023-07-14T17:35:55.991 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T17:35:55.991 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: monmap e1: 1 mons at {a=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0]} removed_ranks: {} 2023-07-14T17:35:55.991 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: fsmap 2023-07-14T17:35:55.991 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: osdmap e1: 0 total, 0 up, 0 in 2023-07-14T17:35:55.992 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 ceph-mon[127465]: mgrmap e1: no daemons active 2023-07-14T17:35:55.992 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:55 smithi088 systemd[1]: Started Ceph mon.a for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:35:59.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:35:59 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/4071237238' entity='client.admin' 2023-07-14T17:36:00.442 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.328+0000 7fb9013bb700 1 Processor -- start 2023-07-14T17:36:00.442 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.328+0000 7fb9013bb700 1 -- start start 2023-07-14T17:36:00.443 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.328+0000 7fb9013bb700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8fc1034c0 0x7fb8fc1038d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:00.443 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.328+0000 7fb9013bb700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc103e10 con 0x7fb8fc1034c0 2023-07-14T17:36:00.443 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.329+0000 7fb8faffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8fc1034c0 0x7fb8fc1038d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:00.443 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.329+0000 7fb8faffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8fc1034c0 0x7fb8fc1038d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40110/0 (socket says 172.21.15.88:40110) 2023-07-14T17:36:00.443 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.329+0000 7fb8faffd700 1 -- 172.21.15.88:0/525434689 learned_addr learned my addr 172.21.15.88:0/525434689 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:00.443 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.329+0000 7fb8faffd700 1 -- 172.21.15.88:0/525434689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc103f50 con 0x7fb8fc1034c0 2023-07-14T17:36:00.444 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.329+0000 7fb8faffd700 1 --2- 172.21.15.88:0/525434689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8fc1034c0 0x7fb8fc1038d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb8f00098a0 tx=0x7fb8f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6558889d99a8c0d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:00.444 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.329+0000 7fb8f9ffb700 1 -- 172.21.15.88:0/525434689 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8f0004160 con 0x7fb8fc1034c0 2023-07-14T17:36:00.444 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.330+0000 7fb8f9ffb700 1 -- 172.21.15.88:0/525434689 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fb8f00042c0 con 0x7fb8fc1034c0 2023-07-14T17:36:00.444 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.330+0000 7fb8f9ffb700 1 -- 172.21.15.88:0/525434689 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8f0011940 con 0x7fb8fc1034c0 2023-07-14T17:36:00.444 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.330+0000 7fb9013bb700 1 -- 172.21.15.88:0/525434689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8fc1034c0 msgr2=0x7fb8fc1038d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:00.445 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.330+0000 7fb9013bb700 1 --2- 172.21.15.88:0/525434689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8fc1034c0 0x7fb8fc1038d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb8f00098a0 tx=0x7fb8f0009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:00.445 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.330+0000 7fb9013bb700 1 -- 172.21.15.88:0/525434689 shutdown_connections 2023-07-14T17:36:00.445 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.330+0000 7fb9013bb700 1 --2- 172.21.15.88:0/525434689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8fc1034c0 0x7fb8fc1038d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:00.445 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.330+0000 7fb9013bb700 1 -- 172.21.15.88:0/525434689 >> 172.21.15.88:0/525434689 conn(0x7fb8fc0feec0 msgr2=0x7fb8fc1012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:00.445 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.330+0000 7fb9013bb700 1 -- 172.21.15.88:0/525434689 shutdown_connections 2023-07-14T17:36:00.445 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.330+0000 7fb9013bb700 1 -- 172.21.15.88:0/525434689 wait complete. 2023-07-14T17:36:00.445 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.331+0000 7fb9013bb700 1 Processor -- start 2023-07-14T17:36:00.446 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.331+0000 7fb9013bb700 1 -- start start 2023-07-14T17:36:00.446 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.331+0000 7fb9013bb700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8fc1034c0 0x7fb8fc198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:00.446 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.331+0000 7fb9013bb700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc199470 con 0x7fb8fc1034c0 2023-07-14T17:36:00.446 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.331+0000 7fb8faffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8fc1034c0 0x7fb8fc198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:00.446 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.331+0000 7fb8faffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8fc1034c0 0x7fb8fc198f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40112/0 (socket says 172.21.15.88:40112) 2023-07-14T17:36:00.446 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.331+0000 7fb8faffd700 1 -- 172.21.15.88:0/4071237238 learned_addr learned my addr 172.21.15.88:0/4071237238 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:00.446 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.331+0000 7fb8faffd700 1 -- 172.21.15.88:0/4071237238 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f0009550 con 0x7fb8fc1034c0 2023-07-14T17:36:00.446 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.332+0000 7fb8faffd700 1 --2- 172.21.15.88:0/4071237238 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8fc1034c0 0x7fb8fc198f30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0000c00 tx=0x7fb8f0011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:00.447 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.332+0000 7fb8e3fff700 1 -- 172.21.15.88:0/4071237238 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8f00036a0 con 0x7fb8fc1034c0 2023-07-14T17:36:00.447 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.332+0000 7fb8e3fff700 1 -- 172.21.15.88:0/4071237238 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fb8f001b440 con 0x7fb8fc1034c0 2023-07-14T17:36:00.447 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.332+0000 7fb9013bb700 1 -- 172.21.15.88:0/4071237238 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8fc199670 con 0x7fb8fc1034c0 2023-07-14T17:36:00.447 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.332+0000 7fb8e3fff700 1 -- 172.21.15.88:0/4071237238 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8f00193f0 con 0x7fb8fc1034c0 2023-07-14T17:36:00.447 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.332+0000 7fb9013bb700 1 -- 172.21.15.88:0/4071237238 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8fc199a90 con 0x7fb8fc1034c0 2023-07-14T17:36:00.447 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.332+0000 7fb8e3fff700 1 -- 172.21.15.88:0/4071237238 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb8f0019930 con 0x7fb8fc1034c0 2023-07-14T17:36:00.447 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.332+0000 7fb8e3fff700 1 -- 172.21.15.88:0/4071237238 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb8f001a3d0 con 0x7fb8fc1034c0 2023-07-14T17:36:00.447 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.333+0000 7fb9013bb700 1 -- 172.21.15.88:0/4071237238 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8fc068520 con 0x7fb8fc1034c0 2023-07-14T17:36:00.448 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.335+0000 7fb8e3fff700 1 -- 172.21.15.88:0/4071237238 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fb8f0029060 con 0x7fb8fc1034c0 2023-07-14T17:36:00.448 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.402+0000 7fb9013bb700 1 -- 172.21.15.88:0/4071237238 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fb8fc19a610 con 0x7fb8fc1034c0 2023-07-14T17:36:00.448 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.405+0000 7fb8e3fff700 1 -- 172.21.15.88:0/4071237238 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fb8f001f070 con 0x7fb8fc1034c0 2023-07-14T17:36:00.448 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.405+0000 7fb9013bb700 1 -- 172.21.15.88:0/4071237238 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8fc1034c0 msgr2=0x7fb8fc198f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:00.448 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.405+0000 7fb9013bb700 1 --2- 172.21.15.88:0/4071237238 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8fc1034c0 0x7fb8fc198f30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb8f0000c00 tx=0x7fb8f0011e70 comp rx=0 tx=0).stop 2023-07-14T17:36:00.448 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.406+0000 7fb9013bb700 1 -- 172.21.15.88:0/4071237238 shutdown_connections 2023-07-14T17:36:00.448 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.406+0000 7fb9013bb700 1 --2- 172.21.15.88:0/4071237238 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8fc1034c0 0x7fb8fc198f30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:00.449 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.406+0000 7fb9013bb700 1 -- 172.21.15.88:0/4071237238 >> 172.21.15.88:0/4071237238 conn(0x7fb8fc0feec0 msgr2=0x7fb8fc0ffbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:00.449 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.406+0000 7fb9013bb700 1 -- 172.21.15.88:0/4071237238 shutdown_connections 2023-07-14T17:36:00.449 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:35:58.406+0000 7fb9013bb700 1 -- 172.21.15.88:0/4071237238 wait complete. 2023-07-14T17:36:00.449 INFO:teuthology.orchestra.run.smithi088.stdout:Wrote config to /etc/ceph/ceph.conf 2023-07-14T17:36:00.778 INFO:teuthology.orchestra.run.smithi088.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-14T17:36:00.778 INFO:teuthology.orchestra.run.smithi088.stdout:Creating mgr... 2023-07-14T17:36:00.779 INFO:teuthology.orchestra.run.smithi088.stdout:Verifying port 9283 ... 2023-07-14T17:36:00.779 INFO:teuthology.orchestra.run.smithi088.stdout:Verifying port 8765 ... 2023-07-14T17:36:01.669 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-b462a556-226c-11ee-9b33-001a4aab830c@mgr.x 2023-07-14T17:36:01.669 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Failed to reset failed state of unit ceph-b462a556-226c-11ee-9b33-001a4aab830c@mgr.x.service: Unit ceph-b462a556-226c-11ee-9b33-001a4aab830c@mgr.x.service not loaded. 2023-07-14T17:36:01.756 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-b462a556-226c-11ee-9b33-001a4aab830c.target.wants/ceph-b462a556-226c-11ee-9b33-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-b462a556-226c-11ee-9b33-001a4aab830c@.service. 2023-07-14T17:36:01.946 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:01 smithi088 systemd[1]: Starting Ceph mgr.x for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:36:02.311 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:02 smithi088 podman[127694]: 2023-07-14T17:36:02.356 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T17:36:02.356 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout disabled 2023-07-14T17:36:02.363 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T17:36:02.364 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout inactive 2023-07-14T17:36:02.364 INFO:teuthology.orchestra.run.smithi088.stdout:firewalld.service is not enabled 2023-07-14T17:36:02.364 INFO:teuthology.orchestra.run.smithi088.stdout:Not possible to enable service . firewalld.service is not available 2023-07-14T17:36:02.371 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T17:36:02.372 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout disabled 2023-07-14T17:36:02.379 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T17:36:02.379 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout inactive 2023-07-14T17:36:02.379 INFO:teuthology.orchestra.run.smithi088.stdout:firewalld.service is not enabled 2023-07-14T17:36:02.379 INFO:teuthology.orchestra.run.smithi088.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-07-14T17:36:02.379 INFO:teuthology.orchestra.run.smithi088.stdout:Waiting for mgr to start... 2023-07-14T17:36:02.380 INFO:teuthology.orchestra.run.smithi088.stdout:Waiting for mgr... 2023-07-14T17:36:02.608 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:02 smithi088 bash[127694]: 7b3ece0f3903c452a7e5674e6c552cf995c6c4b63001371c3a73b60c138a2ad9 2023-07-14T17:36:02.608 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:02 smithi088 systemd[1]: Started Ceph mgr.x for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:36:02.608 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:02 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:02.448+0000 7f4d2975e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T17:36:02.609 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:02 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:02.527+0000 7f4d2975e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T17:36:02.609 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:02 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:02.619+0000 7f4d2975e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T17:36:02.882 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:02 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:02.719+0000 7f4d2975e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T17:36:02.883 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:02 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:02.894+0000 7f4d2975e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T17:36:03.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:03 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2230940664' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T17:36:03.328 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2023-07-14T17:36:03.329 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout { 2023-07-14T17:36:03.329 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "fsid": "b462a556-226c-11ee-9b33-001a4aab830c", 2023-07-14T17:36:03.329 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T17:36:03.329 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T17:36:03.329 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T17:36:03.330 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T17:36:03.330 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:03.330 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T17:36:03.330 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T17:36:03.330 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 0 2023-07-14T17:36:03.330 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2023-07-14T17:36:03.330 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T17:36:03.331 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T17:36:03.331 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2023-07-14T17:36:03.331 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2023-07-14T17:36:03.331 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T17:36:03.331 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T17:36:03.331 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T17:36:03.331 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T17:36:03.331 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:03.332 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T17:36:03.332 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T17:36:03.332 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T17:36:03.332 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T17:36:03.332 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T17:36:03.338 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T17:36:03.338 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T17:36:03.338 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T17:36:03.338 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:03.338 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T17:36:03.338 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T17:36:03.339 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T17:36:03.339 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T17:36:03.339 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T17:36:03.339 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T17:36:03.339 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T17:36:03.339 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T17:36:03.339 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T17:36:03.339 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:03.340 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T17:36:03.340 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T17:36:03.340 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T17:36:03.340 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T17:36:03.340 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:03.340 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T17:36:03.340 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T17:36:03.341 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T17:36:03.341 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T17:36:03.341 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T17:36:03.341 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T17:36:03.341 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T17:36:03.341 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2023-07-14T17:36:03.341 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T17:36:03.341 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:03.342 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T17:36:03.342 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T17:36:03.342 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T17:35:48.898732+0000", 2023-07-14T17:36:03.342 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T17:36:03.342 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:03.342 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T17:36:03.342 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout } 2023-07-14T17:36:03.342 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.904+0000 7fe990210700 1 Processor -- start 2023-07-14T17:36:03.343 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.904+0000 7fe990210700 1 -- start start 2023-07-14T17:36:03.343 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.904+0000 7fe990210700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe988105d30 0x7fe988108110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:03.343 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.904+0000 7fe990210700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe988079d10 con 0x7fe988105d30 2023-07-14T17:36:03.343 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.904+0000 7fe98dfac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe988105d30 0x7fe988108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:03.343 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.904+0000 7fe98dfac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe988105d30 0x7fe988108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40118/0 (socket says 172.21.15.88:40118) 2023-07-14T17:36:03.343 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.904+0000 7fe98dfac700 1 -- 172.21.15.88:0/3750565024 learned_addr learned my addr 172.21.15.88:0/3750565024 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:03.343 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.905+0000 7fe98dfac700 1 -- 172.21.15.88:0/3750565024 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe988079e50 con 0x7fe988105d30 2023-07-14T17:36:03.344 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.905+0000 7fe98dfac700 1 --2- 172.21.15.88:0/3750565024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe988105d30 0x7fe988108110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe9840098a0 tx=0x7fe984009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=106c061786581bca server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:03.344 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.905+0000 7fe98cfaa700 1 -- 172.21.15.88:0/3750565024 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe984004160 con 0x7fe988105d30 2023-07-14T17:36:03.344 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.905+0000 7fe98cfaa700 1 -- 172.21.15.88:0/3750565024 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fe9840042c0 con 0x7fe988105d30 2023-07-14T17:36:03.344 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.905+0000 7fe98cfaa700 1 -- 172.21.15.88:0/3750565024 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe984011940 con 0x7fe988105d30 2023-07-14T17:36:03.344 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.906+0000 7fe990210700 1 -- 172.21.15.88:0/3750565024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe988105d30 msgr2=0x7fe988108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:03.344 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.906+0000 7fe990210700 1 --2- 172.21.15.88:0/3750565024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe988105d30 0x7fe988108110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe9840098a0 tx=0x7fe984009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:03.344 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.906+0000 7fe990210700 1 -- 172.21.15.88:0/3750565024 shutdown_connections 2023-07-14T17:36:03.344 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.906+0000 7fe990210700 1 --2- 172.21.15.88:0/3750565024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe988105d30 0x7fe988108110 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:03.345 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.906+0000 7fe990210700 1 -- 172.21.15.88:0/3750565024 >> 172.21.15.88:0/3750565024 conn(0x7fe9880ff770 msgr2=0x7fe988101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:03.345 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.906+0000 7fe990210700 1 -- 172.21.15.88:0/3750565024 shutdown_connections 2023-07-14T17:36:03.345 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.906+0000 7fe990210700 1 -- 172.21.15.88:0/3750565024 wait complete. 2023-07-14T17:36:03.345 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.906+0000 7fe990210700 1 Processor -- start 2023-07-14T17:36:03.345 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.906+0000 7fe990210700 1 -- start start 2023-07-14T17:36:03.345 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.906+0000 7fe990210700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe988105d30 0x7fe988198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:03.348 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.906+0000 7fe990210700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe988199450 con 0x7fe988105d30 2023-07-14T17:36:03.348 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.907+0000 7fe98dfac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe988105d30 0x7fe988198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:03.349 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.907+0000 7fe98dfac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe988105d30 0x7fe988198f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40120/0 (socket says 172.21.15.88:40120) 2023-07-14T17:36:03.349 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.907+0000 7fe98dfac700 1 -- 172.21.15.88:0/2230940664 learned_addr learned my addr 172.21.15.88:0/2230940664 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:03.349 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.907+0000 7fe98dfac700 1 -- 172.21.15.88:0/2230940664 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe984009550 con 0x7fe988105d30 2023-07-14T17:36:03.350 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.907+0000 7fe98dfac700 1 --2- 172.21.15.88:0/2230940664 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe988105d30 0x7fe988198f10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe984006af0 tx=0x7fe984011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:03.350 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.907+0000 7fe97effd700 1 -- 172.21.15.88:0/2230940664 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe984003bf0 con 0x7fe988105d30 2023-07-14T17:36:03.350 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.907+0000 7fe97effd700 1 -- 172.21.15.88:0/2230940664 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fe98401b8b0 con 0x7fe988105d30 2023-07-14T17:36:03.350 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.907+0000 7fe990210700 1 -- 172.21.15.88:0/2230940664 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9881996b0 con 0x7fe988105d30 2023-07-14T17:36:03.351 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.907+0000 7fe97effd700 1 -- 172.21.15.88:0/2230940664 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe9840198b0 con 0x7fe988105d30 2023-07-14T17:36:03.351 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.907+0000 7fe990210700 1 -- 172.21.15.88:0/2230940664 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe98819a2e0 con 0x7fe988105d30 2023-07-14T17:36:03.351 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.907+0000 7fe97effd700 1 -- 172.21.15.88:0/2230940664 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe9840233f0 con 0x7fe988105d30 2023-07-14T17:36:03.351 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.907+0000 7fe97effd700 1 -- 172.21.15.88:0/2230940664 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe98401a710 con 0x7fe988105d30 2023-07-14T17:36:03.352 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.908+0000 7fe990210700 1 -- 172.21.15.88:0/2230940664 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe988061c70 con 0x7fe988105d30 2023-07-14T17:36:03.352 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.910+0000 7fe97effd700 1 -- 172.21.15.88:0/2230940664 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fe984028060 con 0x7fe988105d30 2023-07-14T17:36:03.352 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.980+0000 7fe990210700 1 -- 172.21.15.88:0/2230940664 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe98819a760 con 0x7fe988105d30 2023-07-14T17:36:03.353 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.981+0000 7fe97effd700 1 -- 172.21.15.88:0/2230940664 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fe984031090 con 0x7fe988105d30 2023-07-14T17:36:03.353 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.982+0000 7fe990210700 1 -- 172.21.15.88:0/2230940664 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe988105d30 msgr2=0x7fe988198f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:03.353 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.982+0000 7fe990210700 1 --2- 172.21.15.88:0/2230940664 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe988105d30 0x7fe988198f10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe984006af0 tx=0x7fe984011a30 comp rx=0 tx=0).stop 2023-07-14T17:36:03.354 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.982+0000 7fe990210700 1 -- 172.21.15.88:0/2230940664 shutdown_connections 2023-07-14T17:36:03.354 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.982+0000 7fe990210700 1 --2- 172.21.15.88:0/2230940664 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe988105d30 0x7fe988198f10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:03.354 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.982+0000 7fe990210700 1 -- 172.21.15.88:0/2230940664 >> 172.21.15.88:0/2230940664 conn(0x7fe9880ff770 msgr2=0x7fe988100510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:03.354 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.982+0000 7fe990210700 1 -- 172.21.15.88:0/2230940664 shutdown_connections 2023-07-14T17:36:03.355 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:02.982+0000 7fe990210700 1 -- 172.21.15.88:0/2230940664 wait complete. 2023-07-14T17:36:03.355 INFO:teuthology.orchestra.run.smithi088.stdout:mgr not available, waiting (1/15)... 2023-07-14T17:36:03.521 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:03 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:03.218+0000 7f4d2975e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T17:36:03.521 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:03 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:03.309+0000 7f4d2975e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T17:36:03.521 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:03 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:03.440+0000 7f4d2975e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T17:36:03.934 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:03 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:03.532+0000 7f4d2975e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T17:36:04.196 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:03 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:03.946+0000 7f4d2975e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T17:36:04.946 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:04 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:04.513+0000 7f4d2975e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T17:36:04.947 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:04 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:04.681+0000 7f4d2975e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T17:36:05.696 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:05 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:05.454+0000 7f4d2975e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T17:36:05.697 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:05 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:05.544+0000 7f4d2975e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T17:36:06.014 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:05 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:05.724+0000 7f4d2975e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T17:36:06.289 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2023-07-14T17:36:06.290 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout { 2023-07-14T17:36:06.290 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "fsid": "b462a556-226c-11ee-9b33-001a4aab830c", 2023-07-14T17:36:06.290 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T17:36:06.290 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T17:36:06.290 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T17:36:06.290 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T17:36:06.290 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:06.291 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T17:36:06.291 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T17:36:06.291 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 0 2023-07-14T17:36:06.291 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2023-07-14T17:36:06.291 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T17:36:06.291 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T17:36:06.291 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2023-07-14T17:36:06.291 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2023-07-14T17:36:06.292 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T17:36:06.292 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T17:36:06.292 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T17:36:06.292 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T17:36:06.292 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:06.292 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T17:36:06.292 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T17:36:06.293 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T17:36:06.293 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T17:36:06.293 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T17:36:06.299 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T17:36:06.299 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T17:36:06.299 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T17:36:06.299 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:06.299 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T17:36:06.300 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T17:36:06.300 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T17:36:06.300 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T17:36:06.300 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T17:36:06.300 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T17:36:06.300 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T17:36:06.300 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T17:36:06.301 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T17:36:06.301 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:06.301 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T17:36:06.301 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T17:36:06.301 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T17:36:06.301 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T17:36:06.301 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:06.302 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T17:36:06.302 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T17:36:06.302 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T17:36:06.302 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T17:36:06.302 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T17:36:06.302 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T17:36:06.302 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T17:36:06.302 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2023-07-14T17:36:06.303 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T17:36:06.303 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:06.303 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T17:36:06.303 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T17:36:06.303 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T17:35:48.898732+0000", 2023-07-14T17:36:06.303 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T17:36:06.303 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:06.303 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T17:36:06.304 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout } 2023-07-14T17:36:06.304 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.845+0000 7f080cf42700 1 Processor -- start 2023-07-14T17:36:06.304 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.845+0000 7f080cf42700 1 -- start start 2023-07-14T17:36:06.304 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.845+0000 7f080cf42700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0808105d30 0x7f0808108110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:06.304 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.845+0000 7f080cf42700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0808079d10 con 0x7f0808105d30 2023-07-14T17:36:06.304 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.845+0000 7f080659c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0808105d30 0x7f0808108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:06.304 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.846+0000 7f080659c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0808105d30 0x7f0808108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40122/0 (socket says 172.21.15.88:40122) 2023-07-14T17:36:06.305 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.846+0000 7f080659c700 1 -- 172.21.15.88:0/828130263 learned_addr learned my addr 172.21.15.88:0/828130263 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:06.305 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.846+0000 7f080659c700 1 -- 172.21.15.88:0/828130263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0808079e50 con 0x7f0808105d30 2023-07-14T17:36:06.305 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.846+0000 7f080659c700 1 --2- 172.21.15.88:0/828130263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0808105d30 0x7f0808108110 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f07f80098a0 tx=0x7f07f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c685f009ec95815 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:06.305 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.846+0000 7f080559a700 1 -- 172.21.15.88:0/828130263 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07f8004160 con 0x7f0808105d30 2023-07-14T17:36:06.305 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.846+0000 7f080559a700 1 -- 172.21.15.88:0/828130263 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f07f80042c0 con 0x7f0808105d30 2023-07-14T17:36:06.306 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.846+0000 7f080559a700 1 -- 172.21.15.88:0/828130263 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07f8011940 con 0x7f0808105d30 2023-07-14T17:36:06.306 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.847+0000 7f080cf42700 1 -- 172.21.15.88:0/828130263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0808105d30 msgr2=0x7f0808108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:06.306 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.847+0000 7f080cf42700 1 --2- 172.21.15.88:0/828130263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0808105d30 0x7f0808108110 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f07f80098a0 tx=0x7f07f8009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:06.306 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.847+0000 7f080cf42700 1 -- 172.21.15.88:0/828130263 shutdown_connections 2023-07-14T17:36:06.306 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.847+0000 7f080cf42700 1 --2- 172.21.15.88:0/828130263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0808105d30 0x7f0808108110 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:06.306 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.847+0000 7f080cf42700 1 -- 172.21.15.88:0/828130263 >> 172.21.15.88:0/828130263 conn(0x7f08080ff770 msgr2=0x7f0808101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:06.306 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.847+0000 7f080cf42700 1 -- 172.21.15.88:0/828130263 shutdown_connections 2023-07-14T17:36:06.307 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.847+0000 7f080cf42700 1 -- 172.21.15.88:0/828130263 wait complete. 2023-07-14T17:36:06.307 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.848+0000 7f080cf42700 1 Processor -- start 2023-07-14T17:36:06.307 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.848+0000 7f080cf42700 1 -- start start 2023-07-14T17:36:06.307 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.848+0000 7f080cf42700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0808105d30 0x7f0808198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:06.307 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.848+0000 7f080cf42700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0808199450 con 0x7f0808105d30 2023-07-14T17:36:06.307 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.848+0000 7f080659c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0808105d30 0x7f0808198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:06.307 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.848+0000 7f080659c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0808105d30 0x7f0808198f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40124/0 (socket says 172.21.15.88:40124) 2023-07-14T17:36:06.307 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.848+0000 7f080659c700 1 -- 172.21.15.88:0/1942045387 learned_addr learned my addr 172.21.15.88:0/1942045387 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:06.308 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.848+0000 7f080659c700 1 -- 172.21.15.88:0/1942045387 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f8009550 con 0x7f0808105d30 2023-07-14T17:36:06.308 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.849+0000 7f080659c700 1 --2- 172.21.15.88:0/1942045387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0808105d30 0x7f0808198f10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f07f8000c00 tx=0x7f07f8011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:06.308 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.849+0000 7f07f77fe700 1 -- 172.21.15.88:0/1942045387 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07f80036a0 con 0x7f0808105d30 2023-07-14T17:36:06.308 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.849+0000 7f080cf42700 1 -- 172.21.15.88:0/1942045387 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0808199650 con 0x7f0808105d30 2023-07-14T17:36:06.308 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.849+0000 7f07f77fe700 1 -- 172.21.15.88:0/1942045387 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f07f801b440 con 0x7f0808105d30 2023-07-14T17:36:06.308 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.849+0000 7f07f77fe700 1 -- 172.21.15.88:0/1942045387 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07f80193f0 con 0x7f0808105d30 2023-07-14T17:36:06.308 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.849+0000 7f080cf42700 1 -- 172.21.15.88:0/1942045387 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080819a2e0 con 0x7f0808105d30 2023-07-14T17:36:06.309 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.849+0000 7f07f77fe700 1 -- 172.21.15.88:0/1942045387 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f07f8019970 con 0x7f0808105d30 2023-07-14T17:36:06.309 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.849+0000 7f07f77fe700 1 -- 172.21.15.88:0/1942045387 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f07f801a3d0 con 0x7f0808105d30 2023-07-14T17:36:06.309 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.850+0000 7f080cf42700 1 -- 172.21.15.88:0/1942045387 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0808199ad0 con 0x7f0808105d30 2023-07-14T17:36:06.309 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.852+0000 7f07f77fe700 1 -- 172.21.15.88:0/1942045387 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f07f8029060 con 0x7f0808105d30 2023-07-14T17:36:06.309 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.924+0000 7f080cf42700 1 -- 172.21.15.88:0/1942045387 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0808061c70 con 0x7f0808105d30 2023-07-14T17:36:06.309 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.924+0000 7f07f77fe700 1 -- 172.21.15.88:0/1942045387 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f07f801f070 con 0x7f0808105d30 2023-07-14T17:36:06.309 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.925+0000 7f080cf42700 1 -- 172.21.15.88:0/1942045387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0808105d30 msgr2=0x7f0808198f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:06.310 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.925+0000 7f080cf42700 1 --2- 172.21.15.88:0/1942045387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0808105d30 0x7f0808198f10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f07f8000c00 tx=0x7f07f8011e70 comp rx=0 tx=0).stop 2023-07-14T17:36:06.310 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.925+0000 7f080cf42700 1 -- 172.21.15.88:0/1942045387 shutdown_connections 2023-07-14T17:36:06.310 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.925+0000 7f080cf42700 1 --2- 172.21.15.88:0/1942045387 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0808105d30 0x7f0808198f10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:06.310 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.925+0000 7f080cf42700 1 -- 172.21.15.88:0/1942045387 >> 172.21.15.88:0/1942045387 conn(0x7f08080ff770 msgr2=0x7f0808100510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:06.310 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.925+0000 7f080cf42700 1 -- 172.21.15.88:0/1942045387 shutdown_connections 2023-07-14T17:36:06.310 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:05.925+0000 7f080cf42700 1 -- 172.21.15.88:0/1942045387 wait complete. 2023-07-14T17:36:06.310 INFO:teuthology.orchestra.run.smithi088.stdout:mgr not available, waiting (2/15)... 2023-07-14T17:36:06.353 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:06 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:06.026+0000 7f4d2975e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T17:36:06.354 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:06 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:06.261+0000 7f4d2975e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T17:36:06.656 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:06 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1942045387' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T17:36:06.656 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:06 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:06.410+0000 7f4d2975e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T17:36:06.946 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:06 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:06.669+0000 7f4d2975e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T17:36:07.659 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:07 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:07.358+0000 7f4d2975e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T17:36:07.660 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:07 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:07.486+0000 7f4d2975e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T17:36:07.946 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:07 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:07.672+0000 7f4d2975e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T17:36:07.946 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:07 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:07.784+0000 7f4d2975e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T17:36:08.446 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:07 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:07.996+0000 7f4d2975e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T17:36:09.196 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:08 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:08.881+0000 7f4d2975e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T17:36:09.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:08 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/52758355' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T17:36:09.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:08 smithi088 ceph-mon[127465]: Activating manager daemon x 2023-07-14T17:36:09.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:08 smithi088 ceph-mon[127465]: mgrmap e2: x(active, starting, since 0.00299738s) 2023-07-14T17:36:09.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:08 smithi088 ceph-mon[127465]: from='mgr.14100 172.21.15.88:0/1989961709' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T17:36:09.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:08 smithi088 ceph-mon[127465]: from='mgr.14100 172.21.15.88:0/1989961709' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T17:36:09.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:08 smithi088 ceph-mon[127465]: from='mgr.14100 172.21.15.88:0/1989961709' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T17:36:09.198 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:08 smithi088 ceph-mon[127465]: from='mgr.14100 172.21.15.88:0/1989961709' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T17:36:09.198 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:08 smithi088 ceph-mon[127465]: from='mgr.14100 172.21.15.88:0/1989961709' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T17:36:09.198 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:08 smithi088 ceph-mon[127465]: Manager daemon x is now available 2023-07-14T17:36:09.246 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2023-07-14T17:36:09.247 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout { 2023-07-14T17:36:09.247 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "fsid": "b462a556-226c-11ee-9b33-001a4aab830c", 2023-07-14T17:36:09.247 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T17:36:09.247 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T17:36:09.247 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T17:36:09.247 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T17:36:09.247 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:09.248 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T17:36:09.248 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T17:36:09.248 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 0 2023-07-14T17:36:09.248 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2023-07-14T17:36:09.248 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T17:36:09.248 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T17:36:09.248 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2023-07-14T17:36:09.249 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2023-07-14T17:36:09.249 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T17:36:09.249 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T17:36:09.249 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T17:36:09.249 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T17:36:09.249 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:09.249 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T17:36:09.250 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T17:36:09.250 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T17:36:09.256 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T17:36:09.256 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T17:36:09.256 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T17:36:09.256 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T17:36:09.256 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T17:36:09.256 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:09.257 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T17:36:09.257 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T17:36:09.257 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T17:36:09.257 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T17:36:09.257 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T17:36:09.257 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T17:36:09.257 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T17:36:09.258 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T17:36:09.258 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T17:36:09.258 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:09.258 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T17:36:09.258 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T17:36:09.258 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T17:36:09.258 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T17:36:09.258 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:09.259 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T17:36:09.259 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T17:36:09.259 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T17:36:09.259 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T17:36:09.259 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T17:36:09.259 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T17:36:09.259 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T17:36:09.259 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2023-07-14T17:36:09.260 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T17:36:09.260 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:09.260 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T17:36:09.260 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T17:36:09.260 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T17:35:48.898732+0000", 2023-07-14T17:36:09.260 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T17:36:09.260 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:09.261 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T17:36:09.261 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout } 2023-07-14T17:36:09.261 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.795+0000 7f928bff6700 1 Processor -- start 2023-07-14T17:36:09.261 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.795+0000 7f928bff6700 1 -- start start 2023-07-14T17:36:09.261 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.796+0000 7f928bff6700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9284105cf0 0x7f92841080d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:09.261 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.796+0000 7f928bff6700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9284079d50 con 0x7f9284105cf0 2023-07-14T17:36:09.261 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.796+0000 7f9289d92700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9284105cf0 0x7f92841080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:09.262 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.796+0000 7f9289d92700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9284105cf0 0x7f92841080d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40126/0 (socket says 172.21.15.88:40126) 2023-07-14T17:36:09.262 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.796+0000 7f9289d92700 1 -- 172.21.15.88:0/1083571986 learned_addr learned my addr 172.21.15.88:0/1083571986 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:09.262 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.796+0000 7f9289d92700 1 -- 172.21.15.88:0/1083571986 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9284079e90 con 0x7f9284105cf0 2023-07-14T17:36:09.262 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.796+0000 7f9289d92700 1 --2- 172.21.15.88:0/1083571986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9284105cf0 0x7f92841080d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f92800098a0 tx=0x7f9280009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d6cd41851fe8a7a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:09.262 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.796+0000 7f9288d90700 1 -- 172.21.15.88:0/1083571986 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9280004160 con 0x7f9284105cf0 2023-07-14T17:36:09.262 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.796+0000 7f9288d90700 1 -- 172.21.15.88:0/1083571986 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f92800042c0 con 0x7f9284105cf0 2023-07-14T17:36:09.262 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.796+0000 7f9288d90700 1 -- 172.21.15.88:0/1083571986 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9280011940 con 0x7f9284105cf0 2023-07-14T17:36:09.262 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.797+0000 7f928bff6700 1 -- 172.21.15.88:0/1083571986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9284105cf0 msgr2=0x7f92841080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:09.263 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.797+0000 7f928bff6700 1 --2- 172.21.15.88:0/1083571986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9284105cf0 0x7f92841080d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f92800098a0 tx=0x7f9280009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:09.263 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.797+0000 7f928bff6700 1 -- 172.21.15.88:0/1083571986 shutdown_connections 2023-07-14T17:36:09.263 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.797+0000 7f928bff6700 1 --2- 172.21.15.88:0/1083571986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9284105cf0 0x7f92841080d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:09.263 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.797+0000 7f928bff6700 1 -- 172.21.15.88:0/1083571986 >> 172.21.15.88:0/1083571986 conn(0x7f92840ff770 msgr2=0x7f9284101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:09.263 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.797+0000 7f928bff6700 1 -- 172.21.15.88:0/1083571986 shutdown_connections 2023-07-14T17:36:09.263 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.797+0000 7f928bff6700 1 -- 172.21.15.88:0/1083571986 wait complete. 2023-07-14T17:36:09.263 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.798+0000 7f928bff6700 1 Processor -- start 2023-07-14T17:36:09.264 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.798+0000 7f928bff6700 1 -- start start 2023-07-14T17:36:09.264 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.798+0000 7f928bff6700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9284105cf0 0x7f9284198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:09.264 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.798+0000 7f928bff6700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92841994b0 con 0x7f9284105cf0 2023-07-14T17:36:09.264 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.798+0000 7f9289d92700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9284105cf0 0x7f9284198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:09.264 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.798+0000 7f9289d92700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9284105cf0 0x7f9284198f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40128/0 (socket says 172.21.15.88:40128) 2023-07-14T17:36:09.264 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.798+0000 7f9289d92700 1 -- 172.21.15.88:0/52758355 learned_addr learned my addr 172.21.15.88:0/52758355 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:09.264 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.798+0000 7f9289d92700 1 -- 172.21.15.88:0/52758355 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9280009550 con 0x7f9284105cf0 2023-07-14T17:36:09.265 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.798+0000 7f9289d92700 1 --2- 172.21.15.88:0/52758355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9284105cf0 0x7f9284198f70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9280006af0 tx=0x7f9280011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:09.265 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.799+0000 7f927affd700 1 -- 172.21.15.88:0/52758355 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9280003bf0 con 0x7f9284105cf0 2023-07-14T17:36:09.265 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.799+0000 7f928bff6700 1 -- 172.21.15.88:0/52758355 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92841996b0 con 0x7f9284105cf0 2023-07-14T17:36:09.265 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.799+0000 7f927affd700 1 -- 172.21.15.88:0/52758355 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f928001b8b0 con 0x7f9284105cf0 2023-07-14T17:36:09.265 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.799+0000 7f927affd700 1 -- 172.21.15.88:0/52758355 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f92800198b0 con 0x7f9284105cf0 2023-07-14T17:36:09.265 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.799+0000 7f928bff6700 1 -- 172.21.15.88:0/52758355 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9284199ad0 con 0x7f9284105cf0 2023-07-14T17:36:09.265 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.799+0000 7f927affd700 1 -- 172.21.15.88:0/52758355 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f92800233f0 con 0x7f9284105cf0 2023-07-14T17:36:09.266 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.799+0000 7f927affd700 1 -- 172.21.15.88:0/52758355 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f928001a760 con 0x7f9284105cf0 2023-07-14T17:36:09.266 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.800+0000 7f928bff6700 1 -- 172.21.15.88:0/52758355 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9284109280 con 0x7f9284105cf0 2023-07-14T17:36:09.266 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.802+0000 7f927affd700 1 -- 172.21.15.88:0/52758355 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f9280028060 con 0x7f9284105cf0 2023-07-14T17:36:09.266 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.875+0000 7f928bff6700 1 -- 172.21.15.88:0/52758355 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9284109470 con 0x7f9284105cf0 2023-07-14T17:36:09.266 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.875+0000 7f927affd700 1 -- 172.21.15.88:0/52758355 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f928001bd70 con 0x7f9284105cf0 2023-07-14T17:36:09.266 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.876+0000 7f928bff6700 1 -- 172.21.15.88:0/52758355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9284105cf0 msgr2=0x7f9284198f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:09.266 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.876+0000 7f928bff6700 1 --2- 172.21.15.88:0/52758355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9284105cf0 0x7f9284198f70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9280006af0 tx=0x7f9280011a30 comp rx=0 tx=0).stop 2023-07-14T17:36:09.266 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.876+0000 7f928bff6700 1 -- 172.21.15.88:0/52758355 shutdown_connections 2023-07-14T17:36:09.267 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.876+0000 7f928bff6700 1 --2- 172.21.15.88:0/52758355 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9284105cf0 0x7f9284198f70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:09.267 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.876+0000 7f928bff6700 1 -- 172.21.15.88:0/52758355 >> 172.21.15.88:0/52758355 conn(0x7f92840ff770 msgr2=0x7f9284100510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:09.267 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.876+0000 7f928bff6700 1 -- 172.21.15.88:0/52758355 shutdown_connections 2023-07-14T17:36:09.267 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:08.876+0000 7f928bff6700 1 -- 172.21.15.88:0/52758355 wait complete. 2023-07-14T17:36:09.267 INFO:teuthology.orchestra.run.smithi088.stdout:mgr not available, waiting (3/15)... 2023-07-14T17:36:10.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:09 smithi088 ceph-mon[127465]: from='mgr.14100 172.21.15.88:0/1989961709' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T17:36:10.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:09 smithi088 ceph-mon[127465]: from='mgr.14100 172.21.15.88:0/1989961709' entity='mgr.x' 2023-07-14T17:36:10.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:09 smithi088 ceph-mon[127465]: from='mgr.14100 172.21.15.88:0/1989961709' entity='mgr.x' 2023-07-14T17:36:10.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:09 smithi088 ceph-mon[127465]: from='mgr.14100 172.21.15.88:0/1989961709' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T17:36:10.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:09 smithi088 ceph-mon[127465]: from='mgr.14100 172.21.15.88:0/1989961709' entity='mgr.x' 2023-07-14T17:36:10.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:09 smithi088 ceph-mon[127465]: mgrmap e3: x(active, since 1.00687s) 2023-07-14T17:36:12.400 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2023-07-14T17:36:12.400 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout { 2023-07-14T17:36:12.400 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "fsid": "b462a556-226c-11ee-9b33-001a4aab830c", 2023-07-14T17:36:12.400 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T17:36:12.401 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T17:36:12.401 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T17:36:12.401 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T17:36:12.401 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:12.401 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T17:36:12.401 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T17:36:12.402 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 0 2023-07-14T17:36:12.402 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2023-07-14T17:36:12.402 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T17:36:12.402 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T17:36:12.402 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2023-07-14T17:36:12.402 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2023-07-14T17:36:12.402 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T17:36:12.402 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T17:36:12.403 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T17:36:12.403 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T17:36:12.403 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:12.403 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T17:36:12.403 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T17:36:12.403 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T17:36:12.403 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T17:36:12.403 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T17:36:12.410 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T17:36:12.410 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T17:36:12.410 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T17:36:12.410 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:12.410 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T17:36:12.410 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T17:36:12.411 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T17:36:12.411 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T17:36:12.411 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T17:36:12.411 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T17:36:12.411 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T17:36:12.411 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T17:36:12.411 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T17:36:12.411 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:12.412 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T17:36:12.412 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T17:36:12.412 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T17:36:12.412 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T17:36:12.412 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:12.412 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T17:36:12.412 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T17:36:12.413 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T17:36:12.413 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T17:36:12.413 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T17:36:12.413 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T17:36:12.413 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T17:36:12.413 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ], 2023-07-14T17:36:12.414 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T17:36:12.414 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:12.414 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T17:36:12.414 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T17:36:12.414 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T17:35:48.898732+0000", 2023-07-14T17:36:12.414 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T17:36:12.414 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout }, 2023-07-14T17:36:12.414 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T17:36:12.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout } 2023-07-14T17:36:12.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.732+0000 7f9369024700 1 Processor -- start 2023-07-14T17:36:12.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.732+0000 7f9369024700 1 -- start start 2023-07-14T17:36:12.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.732+0000 7f9369024700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9364103b50 0x7f9364103f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:12.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.732+0000 7f9369024700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9364079cf0 con 0x7f9364103b50 2023-07-14T17:36:12.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.732+0000 7f9362d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9364103b50 0x7f9364103f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:12.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.733+0000 7f9362d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9364103b50 0x7f9364103f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40154/0 (socket says 172.21.15.88:40154) 2023-07-14T17:36:12.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.733+0000 7f9362d9d700 1 -- 172.21.15.88:0/2971924550 learned_addr learned my addr 172.21.15.88:0/2971924550 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:12.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.733+0000 7f9362d9d700 1 -- 172.21.15.88:0/2971924550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9364079e30 con 0x7f9364103b50 2023-07-14T17:36:12.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.733+0000 7f9362d9d700 1 --2- 172.21.15.88:0/2971924550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9364103b50 0x7f9364103f60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f93580098a0 tx=0x7f9358009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ba074e5185a937 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:12.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.733+0000 7f9361d9b700 1 -- 172.21.15.88:0/2971924550 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9358004160 con 0x7f9364103b50 2023-07-14T17:36:12.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.733+0000 7f9361d9b700 1 -- 172.21.15.88:0/2971924550 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f93580042c0 con 0x7f9364103b50 2023-07-14T17:36:12.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.733+0000 7f9361d9b700 1 -- 172.21.15.88:0/2971924550 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9358011940 con 0x7f9364103b50 2023-07-14T17:36:12.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.734+0000 7f9369024700 1 -- 172.21.15.88:0/2971924550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9364103b50 msgr2=0x7f9364103f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:12.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.734+0000 7f9369024700 1 --2- 172.21.15.88:0/2971924550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9364103b50 0x7f9364103f60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f93580098a0 tx=0x7f9358009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:12.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.734+0000 7f9369024700 1 -- 172.21.15.88:0/2971924550 shutdown_connections 2023-07-14T17:36:12.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.734+0000 7f9369024700 1 --2- 172.21.15.88:0/2971924550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9364103b50 0x7f9364103f60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:12.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.734+0000 7f9369024700 1 -- 172.21.15.88:0/2971924550 >> 172.21.15.88:0/2971924550 conn(0x7f93640ff7a0 msgr2=0x7f9364101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:12.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.734+0000 7f9369024700 1 -- 172.21.15.88:0/2971924550 shutdown_connections 2023-07-14T17:36:12.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.734+0000 7f9369024700 1 -- 172.21.15.88:0/2971924550 wait complete. 2023-07-14T17:36:12.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.734+0000 7f9369024700 1 Processor -- start 2023-07-14T17:36:12.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.735+0000 7f9369024700 1 -- start start 2023-07-14T17:36:12.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.735+0000 7f9369024700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9364103b50 0x7f936419b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:12.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.735+0000 7f9369024700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936419b6e0 con 0x7f9364103b50 2023-07-14T17:36:12.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.735+0000 7f9362d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9364103b50 0x7f936419b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:12.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.735+0000 7f9362d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9364103b50 0x7f936419b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40156/0 (socket says 172.21.15.88:40156) 2023-07-14T17:36:12.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.735+0000 7f9362d9d700 1 -- 172.21.15.88:0/160664570 learned_addr learned my addr 172.21.15.88:0/160664570 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:12.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.735+0000 7f9362d9d700 1 -- 172.21.15.88:0/160664570 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9358009550 con 0x7f9364103b50 2023-07-14T17:36:12.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.735+0000 7f9362d9d700 1 --2- 172.21.15.88:0/160664570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9364103b50 0x7f936419b1a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9358006af0 tx=0x7f9358011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:12.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.735+0000 7f934bfff700 1 -- 172.21.15.88:0/160664570 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9358003bf0 con 0x7f9364103b50 2023-07-14T17:36:12.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.735+0000 7f9369024700 1 -- 172.21.15.88:0/160664570 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f936419b8e0 con 0x7f9364103b50 2023-07-14T17:36:12.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.735+0000 7f934bfff700 1 -- 172.21.15.88:0/160664570 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f935801b8b0 con 0x7f9364103b50 2023-07-14T17:36:12.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.735+0000 7f934bfff700 1 -- 172.21.15.88:0/160664570 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93580198b0 con 0x7f9364103b50 2023-07-14T17:36:12.420 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.735+0000 7f9369024700 1 -- 172.21.15.88:0/160664570 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f936419bd00 con 0x7f9364103b50 2023-07-14T17:36:12.420 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.736+0000 7f934bfff700 1 -- 172.21.15.88:0/160664570 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f9358019b70 con 0x7f9364103b50 2023-07-14T17:36:12.420 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.736+0000 7f934bfff700 1 --2- 172.21.15.88:0/160664570 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f934c038670 0x7f934c03ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:12.421 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.736+0000 7f9369024700 1 -- 172.21.15.88:0/160664570 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93641050b0 con 0x7f9364103b50 2023-07-14T17:36:12.421 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.736+0000 7f934bfff700 1 -- 172.21.15.88:0/160664570 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f935804c9d0 con 0x7f9364103b50 2023-07-14T17:36:12.421 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.737+0000 7f936259c700 1 --2- 172.21.15.88:0/160664570 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f934c038670 0x7f934c03ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:12.422 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.737+0000 7f936259c700 1 --2- 172.21.15.88:0/160664570 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f934c038670 0x7f934c03ab20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9354006fd0 tx=0x7f9354006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:12.422 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:11.741+0000 7f934bfff700 1 -- 172.21.15.88:0/160664570 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f93580189a0 con 0x7f9364103b50 2023-07-14T17:36:12.422 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.000+0000 7f9369024700 1 -- 172.21.15.88:0/160664570 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9364105240 con 0x7f9364103b50 2023-07-14T17:36:12.422 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.000+0000 7f934bfff700 1 -- 172.21.15.88:0/160664570 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f9358018350 con 0x7f9364103b50 2023-07-14T17:36:12.423 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.002+0000 7f9369024700 1 -- 172.21.15.88:0/160664570 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f934c038670 msgr2=0x7f934c03ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:12.423 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.002+0000 7f9369024700 1 --2- 172.21.15.88:0/160664570 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f934c038670 0x7f934c03ab20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9354006fd0 tx=0x7f9354006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:12.423 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.002+0000 7f9369024700 1 -- 172.21.15.88:0/160664570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9364103b50 msgr2=0x7f936419b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:12.423 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.002+0000 7f9369024700 1 --2- 172.21.15.88:0/160664570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9364103b50 0x7f936419b1a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9358006af0 tx=0x7f9358011a30 comp rx=0 tx=0).stop 2023-07-14T17:36:12.424 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.002+0000 7f9369024700 1 -- 172.21.15.88:0/160664570 shutdown_connections 2023-07-14T17:36:12.424 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.002+0000 7f9369024700 1 --2- 172.21.15.88:0/160664570 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f934c038670 0x7f934c03ab20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:12.424 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.002+0000 7f9369024700 1 --2- 172.21.15.88:0/160664570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9364103b50 0x7f936419b1a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:12.425 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.002+0000 7f9369024700 1 -- 172.21.15.88:0/160664570 >> 172.21.15.88:0/160664570 conn(0x7f93640ff7a0 msgr2=0x7f936418f710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:12.425 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.003+0000 7f9369024700 1 -- 172.21.15.88:0/160664570 shutdown_connections 2023-07-14T17:36:12.425 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.003+0000 7f9369024700 1 -- 172.21.15.88:0/160664570 wait complete. 2023-07-14T17:36:12.425 INFO:teuthology.orchestra.run.smithi088.stdout:mgr is available 2023-07-14T17:36:12.623 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:12 smithi088 ceph-mon[127465]: mgrmap e4: x(active, since 2s) 2023-07-14T17:36:12.623 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:12 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/160664570' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T17:36:13.446 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2023-07-14T17:36:13.446 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout [global] 2023-07-14T17:36:13.447 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout fsid = b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:36:13.447 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-14T17:36:13.447 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-14T17:36:13.447 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-14T17:36:13.447 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-14T17:36:13.447 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2023-07-14T17:36:13.448 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-14T17:36:13.448 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-14T17:36:13.448 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 2023-07-14T17:36:13.448 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout [osd] 2023-07-14T17:36:13.448 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-14T17:36:13.448 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-14T17:36:13.448 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-14T17:36:13.449 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-14T17:36:13.449 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-14T17:36:13.449 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.872+0000 7f6f609e0700 1 Processor -- start 2023-07-14T17:36:13.449 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.872+0000 7f6f609e0700 1 -- start start 2023-07-14T17:36:13.449 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.873+0000 7f6f609e0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f5c079cd0 0x7f6f5c10a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:13.449 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.873+0000 7f6f609e0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c105b70 con 0x7f6f5c079cd0 2023-07-14T17:36:13.449 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.873+0000 7f6f5a59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f5c079cd0 0x7f6f5c10a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:13.449 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.873+0000 7f6f5a59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f5c079cd0 0x7f6f5c10a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40160/0 (socket says 172.21.15.88:40160) 2023-07-14T17:36:13.450 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.873+0000 7f6f5a59c700 1 -- 172.21.15.88:0/712979024 learned_addr learned my addr 172.21.15.88:0/712979024 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:13.450 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.873+0000 7f6f5a59c700 1 -- 172.21.15.88:0/712979024 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5c10a6c0 con 0x7f6f5c079cd0 2023-07-14T17:36:13.450 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.873+0000 7f6f5a59c700 1 --2- 172.21.15.88:0/712979024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f5c079cd0 0x7f6f5c10a180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6f440098a0 tx=0x7f6f44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff80790899166fd2 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:13.450 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.873+0000 7f6f5959a700 1 -- 172.21.15.88:0/712979024 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f44004160 con 0x7f6f5c079cd0 2023-07-14T17:36:13.450 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.873+0000 7f6f5959a700 1 -- 172.21.15.88:0/712979024 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f6f440042c0 con 0x7f6f5c079cd0 2023-07-14T17:36:13.450 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.873+0000 7f6f5959a700 1 -- 172.21.15.88:0/712979024 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f44011940 con 0x7f6f5c079cd0 2023-07-14T17:36:13.450 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.874+0000 7f6f609e0700 1 -- 172.21.15.88:0/712979024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f5c079cd0 msgr2=0x7f6f5c10a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:13.451 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.874+0000 7f6f609e0700 1 --2- 172.21.15.88:0/712979024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f5c079cd0 0x7f6f5c10a180 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6f440098a0 tx=0x7f6f44009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:13.451 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.874+0000 7f6f609e0700 1 -- 172.21.15.88:0/712979024 shutdown_connections 2023-07-14T17:36:13.451 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.874+0000 7f6f609e0700 1 --2- 172.21.15.88:0/712979024 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f5c079cd0 0x7f6f5c10a180 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:13.451 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.874+0000 7f6f609e0700 1 -- 172.21.15.88:0/712979024 >> 172.21.15.88:0/712979024 conn(0x7f6f5c0ff770 msgr2=0x7f6f5c101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:13.451 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.874+0000 7f6f609e0700 1 -- 172.21.15.88:0/712979024 shutdown_connections 2023-07-14T17:36:13.451 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.874+0000 7f6f609e0700 1 -- 172.21.15.88:0/712979024 wait complete. 2023-07-14T17:36:13.451 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.875+0000 7f6f609e0700 1 Processor -- start 2023-07-14T17:36:13.451 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.875+0000 7f6f609e0700 1 -- start start 2023-07-14T17:36:13.452 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.875+0000 7f6f609e0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f5c19b150 0x7f6f5c19b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:13.452 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.875+0000 7f6f609e0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c19baa0 con 0x7f6f5c19b150 2023-07-14T17:36:13.452 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.875+0000 7f6f5a59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f5c19b150 0x7f6f5c19b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:13.452 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.875+0000 7f6f5a59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f5c19b150 0x7f6f5c19b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40162/0 (socket says 172.21.15.88:40162) 2023-07-14T17:36:13.452 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.875+0000 7f6f5a59c700 1 -- 172.21.15.88:0/2479054187 learned_addr learned my addr 172.21.15.88:0/2479054187 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:13.452 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.875+0000 7f6f5a59c700 1 -- 172.21.15.88:0/2479054187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f44009550 con 0x7f6f5c19b150 2023-07-14T17:36:13.452 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.875+0000 7f6f5a59c700 1 --2- 172.21.15.88:0/2479054187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f5c19b150 0x7f6f5c19b560 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6f44006af0 tx=0x7f6f44011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:13.453 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.876+0000 7f6f537fe700 1 -- 172.21.15.88:0/2479054187 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f44003bf0 con 0x7f6f5c19b150 2023-07-14T17:36:13.453 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.876+0000 7f6f537fe700 1 -- 172.21.15.88:0/2479054187 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f6f4401b8b0 con 0x7f6f5c19b150 2023-07-14T17:36:13.453 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.876+0000 7f6f609e0700 1 -- 172.21.15.88:0/2479054187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5c19bca0 con 0x7f6f5c19b150 2023-07-14T17:36:13.453 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.876+0000 7f6f537fe700 1 -- 172.21.15.88:0/2479054187 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6f440198b0 con 0x7f6f5c19b150 2023-07-14T17:36:13.454 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.876+0000 7f6f609e0700 1 -- 172.21.15.88:0/2479054187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5c19c730 con 0x7f6f5c19b150 2023-07-14T17:36:13.454 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.876+0000 7f6f537fe700 1 -- 172.21.15.88:0/2479054187 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f6f4401ba20 con 0x7f6f5c19b150 2023-07-14T17:36:13.454 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.876+0000 7f6f609e0700 1 -- 172.21.15.88:0/2479054187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5c061c70 con 0x7f6f5c19b150 2023-07-14T17:36:13.454 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.877+0000 7f6f537fe700 1 --2- 172.21.15.88:0/2479054187 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f6f48038670 0x7f6f4803ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:13.454 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.877+0000 7f6f537fe700 1 -- 172.21.15.88:0/2479054187 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6f4404c970 con 0x7f6f5c19b150 2023-07-14T17:36:13.455 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.877+0000 7f6f59d9b700 1 --2- 172.21.15.88:0/2479054187 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f6f48038670 0x7f6f4803ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:13.455 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.877+0000 7f6f59d9b700 1 --2- 172.21.15.88:0/2479054187 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f6f48038670 0x7f6f4803ab20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c006fd0 tx=0x7f6f4c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:13.455 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:12.881+0000 7f6f537fe700 1 -- 172.21.15.88:0/2479054187 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6f44018530 con 0x7f6f5c19b150 2023-07-14T17:36:13.455 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.065+0000 7f6f609e0700 1 -- 172.21.15.88:0/2479054187 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f6f5c068520 con 0x7f6f5c19b150 2023-07-14T17:36:13.455 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.066+0000 7f6f537fe700 1 -- 172.21.15.88:0/2479054187 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f6f4401f250 con 0x7f6f5c19b150 2023-07-14T17:36:13.455 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.068+0000 7f6f609e0700 1 -- 172.21.15.88:0/2479054187 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f6f48038670 msgr2=0x7f6f4803ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:13.455 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.068+0000 7f6f609e0700 1 --2- 172.21.15.88:0/2479054187 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f6f48038670 0x7f6f4803ab20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6f4c006fd0 tx=0x7f6f4c006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:13.456 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.068+0000 7f6f609e0700 1 -- 172.21.15.88:0/2479054187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f5c19b150 msgr2=0x7f6f5c19b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:13.456 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.068+0000 7f6f609e0700 1 --2- 172.21.15.88:0/2479054187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f5c19b150 0x7f6f5c19b560 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6f44006af0 tx=0x7f6f44011a30 comp rx=0 tx=0).stop 2023-07-14T17:36:13.456 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.068+0000 7f6f609e0700 1 -- 172.21.15.88:0/2479054187 shutdown_connections 2023-07-14T17:36:13.456 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.068+0000 7f6f609e0700 1 --2- 172.21.15.88:0/2479054187 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f6f48038670 0x7f6f4803ab20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:13.456 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.068+0000 7f6f609e0700 1 --2- 172.21.15.88:0/2479054187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f5c19b150 0x7f6f5c19b560 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:13.456 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.068+0000 7f6f609e0700 1 -- 172.21.15.88:0/2479054187 >> 172.21.15.88:0/2479054187 conn(0x7f6f5c0ff770 msgr2=0x7f6f5c100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:13.457 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.068+0000 7f6f609e0700 1 -- 172.21.15.88:0/2479054187 shutdown_connections 2023-07-14T17:36:13.457 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.068+0000 7f6f609e0700 1 -- 172.21.15.88:0/2479054187 wait complete. 2023-07-14T17:36:13.457 INFO:teuthology.orchestra.run.smithi088.stdout:Enabling cephadm module... 2023-07-14T17:36:13.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:13 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2479054187' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-14T17:36:14.933 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:14 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/4151008472' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-14T17:36:14.933 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:14 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: ignoring --setuser ceph since I am not root 2023-07-14T17:36:14.933 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:14 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: ignoring --setgroup ceph since I am not root 2023-07-14T17:36:14.933 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:14 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:14.773+0000 7f757aaae200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T17:36:14.934 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:14 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:14.855+0000 7f757aaae200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T17:36:14.986 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.872+0000 7fbe87fb0700 1 Processor -- start 2023-07-14T17:36:14.986 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.872+0000 7fbe87fb0700 1 -- start start 2023-07-14T17:36:14.986 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.872+0000 7fbe87fb0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe80079cd0 0x7fbe8010a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:14.986 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.872+0000 7fbe87fb0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe80105b70 con 0x7fbe80079cd0 2023-07-14T17:36:14.987 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.872+0000 7fbe85d4c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe80079cd0 0x7fbe8010a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:14.987 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.872+0000 7fbe85d4c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe80079cd0 0x7fbe8010a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40166/0 (socket says 172.21.15.88:40166) 2023-07-14T17:36:14.987 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.872+0000 7fbe85d4c700 1 -- 172.21.15.88:0/2542027286 learned_addr learned my addr 172.21.15.88:0/2542027286 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:14.987 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.873+0000 7fbe85d4c700 1 -- 172.21.15.88:0/2542027286 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8010a6c0 con 0x7fbe80079cd0 2023-07-14T17:36:14.987 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.873+0000 7fbe85d4c700 1 --2- 172.21.15.88:0/2542027286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe80079cd0 0x7fbe8010a180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbe700098a0 tx=0x7fbe70009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d51f9c8b48420ded server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:14.987 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.873+0000 7fbe84d4a700 1 -- 172.21.15.88:0/2542027286 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe70004160 con 0x7fbe80079cd0 2023-07-14T17:36:14.987 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.873+0000 7fbe84d4a700 1 -- 172.21.15.88:0/2542027286 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fbe700042c0 con 0x7fbe80079cd0 2023-07-14T17:36:14.988 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.873+0000 7fbe84d4a700 1 -- 172.21.15.88:0/2542027286 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe70011940 con 0x7fbe80079cd0 2023-07-14T17:36:14.988 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.874+0000 7fbe87fb0700 1 -- 172.21.15.88:0/2542027286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe80079cd0 msgr2=0x7fbe8010a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:14.988 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.874+0000 7fbe87fb0700 1 --2- 172.21.15.88:0/2542027286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe80079cd0 0x7fbe8010a180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbe700098a0 tx=0x7fbe70009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:14.991 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.874+0000 7fbe87fb0700 1 -- 172.21.15.88:0/2542027286 shutdown_connections 2023-07-14T17:36:14.991 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.874+0000 7fbe87fb0700 1 --2- 172.21.15.88:0/2542027286 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe80079cd0 0x7fbe8010a180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:14.991 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.874+0000 7fbe87fb0700 1 -- 172.21.15.88:0/2542027286 >> 172.21.15.88:0/2542027286 conn(0x7fbe800ff750 msgr2=0x7fbe80101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:14.991 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.874+0000 7fbe87fb0700 1 -- 172.21.15.88:0/2542027286 shutdown_connections 2023-07-14T17:36:14.991 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.874+0000 7fbe87fb0700 1 -- 172.21.15.88:0/2542027286 wait complete. 2023-07-14T17:36:14.992 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.874+0000 7fbe87fb0700 1 Processor -- start 2023-07-14T17:36:14.992 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.874+0000 7fbe87fb0700 1 -- start start 2023-07-14T17:36:14.992 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.874+0000 7fbe87fb0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe8019b0f0 0x7fbe8019b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:14.992 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.874+0000 7fbe87fb0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8019ba40 con 0x7fbe8019b0f0 2023-07-14T17:36:14.992 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.874+0000 7fbe85d4c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe8019b0f0 0x7fbe8019b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:14.992 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.874+0000 7fbe85d4c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe8019b0f0 0x7fbe8019b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40168/0 (socket says 172.21.15.88:40168) 2023-07-14T17:36:14.992 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.875+0000 7fbe85d4c700 1 -- 172.21.15.88:0/4151008472 learned_addr learned my addr 172.21.15.88:0/4151008472 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:14.992 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.875+0000 7fbe85d4c700 1 -- 172.21.15.88:0/4151008472 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe70009550 con 0x7fbe8019b0f0 2023-07-14T17:36:14.993 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.875+0000 7fbe85d4c700 1 --2- 172.21.15.88:0/4151008472 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe8019b0f0 0x7fbe8019b500 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbe70000c00 tx=0x7fbe70011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:14.993 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.875+0000 7fbe76ffd700 1 -- 172.21.15.88:0/4151008472 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe700036a0 con 0x7fbe8019b0f0 2023-07-14T17:36:14.993 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.875+0000 7fbe76ffd700 1 -- 172.21.15.88:0/4151008472 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fbe7001b440 con 0x7fbe8019b0f0 2023-07-14T17:36:14.993 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.875+0000 7fbe87fb0700 1 -- 172.21.15.88:0/4151008472 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe8019bc40 con 0x7fbe8019b0f0 2023-07-14T17:36:14.993 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.875+0000 7fbe76ffd700 1 -- 172.21.15.88:0/4151008472 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe700193f0 con 0x7fbe8019b0f0 2023-07-14T17:36:14.993 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.875+0000 7fbe87fb0700 1 -- 172.21.15.88:0/4151008472 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe8019c6d0 con 0x7fbe8019b0f0 2023-07-14T17:36:14.993 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.876+0000 7fbe76ffd700 1 -- 172.21.15.88:0/4151008472 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7fbe70019630 con 0x7fbe8019b0f0 2023-07-14T17:36:14.994 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.876+0000 7fbe87fb0700 1 -- 172.21.15.88:0/4151008472 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe80061c70 con 0x7fbe8019b0f0 2023-07-14T17:36:14.994 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.876+0000 7fbe76ffd700 1 --2- 172.21.15.88:0/4151008472 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7fbe6c0386c0 0x7fbe6c03ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:14.994 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.876+0000 7fbe76ffd700 1 -- 172.21.15.88:0/4151008472 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fbe7004c780 con 0x7fbe8019b0f0 2023-07-14T17:36:14.994 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.876+0000 7fbe8554b700 1 --2- 172.21.15.88:0/4151008472 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7fbe6c0386c0 0x7fbe6c03ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:14.994 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.877+0000 7fbe8554b700 1 --2- 172.21.15.88:0/4151008472 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7fbe6c0386c0 0x7fbe6c03ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c006fd0 tx=0x7fbe7c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:14.994 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:13.881+0000 7fbe76ffd700 1 -- 172.21.15.88:0/4151008472 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbe70018d30 con 0x7fbe8019b0f0 2023-07-14T17:36:14.994 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:14.117+0000 7fbe87fb0700 1 -- 172.21.15.88:0/4151008472 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fbe80068520 con 0x7fbe8019b0f0 2023-07-14T17:36:14.995 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:14.611+0000 7fbe76ffd700 1 -- 172.21.15.88:0/4151008472 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fbe7001a3d0 con 0x7fbe8019b0f0 2023-07-14T17:36:14.995 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:14.612+0000 7fbe76ffd700 1 -- 172.21.15.88:0/4151008472 <== mon.0 v2:172.21.15.88:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fbe7002c750 con 0x7fbe8019b0f0 2023-07-14T17:36:14.995 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:14.615+0000 7fbe87fb0700 1 -- 172.21.15.88:0/4151008472 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7fbe6c0386c0 msgr2=0x7fbe6c03ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:14.995 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:14.615+0000 7fbe87fb0700 1 --2- 172.21.15.88:0/4151008472 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7fbe6c0386c0 0x7fbe6c03ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c006fd0 tx=0x7fbe7c006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:14.995 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:14.615+0000 7fbe87fb0700 1 -- 172.21.15.88:0/4151008472 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe8019b0f0 msgr2=0x7fbe8019b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:14.995 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:14.615+0000 7fbe87fb0700 1 --2- 172.21.15.88:0/4151008472 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe8019b0f0 0x7fbe8019b500 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbe70000c00 tx=0x7fbe70011e70 comp rx=0 tx=0).stop 2023-07-14T17:36:14.995 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:14.615+0000 7fbe87fb0700 1 -- 172.21.15.88:0/4151008472 shutdown_connections 2023-07-14T17:36:14.995 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:14.615+0000 7fbe87fb0700 1 --2- 172.21.15.88:0/4151008472 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7fbe6c0386c0 0x7fbe6c03ab70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:14.996 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:14.615+0000 7fbe87fb0700 1 --2- 172.21.15.88:0/4151008472 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe8019b0f0 0x7fbe8019b500 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:14.996 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:14.615+0000 7fbe87fb0700 1 -- 172.21.15.88:0/4151008472 >> 172.21.15.88:0/4151008472 conn(0x7fbe800ff750 msgr2=0x7fbe801004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:14.996 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:14.615+0000 7fbe87fb0700 1 -- 172.21.15.88:0/4151008472 shutdown_connections 2023-07-14T17:36:14.996 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:14.615+0000 7fbe87fb0700 1 -- 172.21.15.88:0/4151008472 wait complete. 2023-07-14T17:36:15.196 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:14 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:14.945+0000 7f757aaae200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T17:36:15.196 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:15 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:15.049+0000 7f757aaae200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T17:36:15.530 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:15 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:15.222+0000 7f757aaae200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T17:36:15.855 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:15 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/4151008472' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-14T17:36:15.855 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:15 smithi088 ceph-mon[127465]: mgrmap e5: x(active, since 5s) 2023-07-14T17:36:15.856 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:15 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:15.543+0000 7f757aaae200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T17:36:15.856 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:15 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:15.634+0000 7f757aaae200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T17:36:15.856 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:15 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:15.767+0000 7f757aaae200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T17:36:15.856 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:15 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:15.868+0000 7f757aaae200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T17:36:16.140 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout { 2023-07-14T17:36:16.140 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-07-14T17:36:16.140 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T17:36:16.141 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-14T17:36:16.141 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-14T17:36:16.141 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout } 2023-07-14T17:36:16.141 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.489+0000 7faa99fef700 1 Processor -- start 2023-07-14T17:36:16.141 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.489+0000 7faa99fef700 1 -- start start 2023-07-14T17:36:16.141 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.489+0000 7faa99fef700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94105cf0 0x7faa941080d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:16.141 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.489+0000 7faa99fef700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa94079d50 con 0x7faa94105cf0 2023-07-14T17:36:16.142 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.489+0000 7faa937fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94105cf0 0x7faa941080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:16.142 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.489+0000 7faa937fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94105cf0 0x7faa941080d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40182/0 (socket says 172.21.15.88:40182) 2023-07-14T17:36:16.142 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.489+0000 7faa937fe700 1 -- 172.21.15.88:0/2449942208 learned_addr learned my addr 172.21.15.88:0/2449942208 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:16.142 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.490+0000 7faa937fe700 1 -- 172.21.15.88:0/2449942208 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa94079e90 con 0x7faa94105cf0 2023-07-14T17:36:16.142 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.490+0000 7faa937fe700 1 --2- 172.21.15.88:0/2449942208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94105cf0 0x7faa941080d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faa840098a0 tx=0x7faa84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8ffe0dbca0b1315 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:16.142 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.490+0000 7faa927fc700 1 -- 172.21.15.88:0/2449942208 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa84004160 con 0x7faa94105cf0 2023-07-14T17:36:16.142 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.490+0000 7faa927fc700 1 -- 172.21.15.88:0/2449942208 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7faa840042c0 con 0x7faa94105cf0 2023-07-14T17:36:16.143 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.490+0000 7faa927fc700 1 -- 172.21.15.88:0/2449942208 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa84011940 con 0x7faa94105cf0 2023-07-14T17:36:16.143 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.490+0000 7faa99fef700 1 -- 172.21.15.88:0/2449942208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94105cf0 msgr2=0x7faa941080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:16.144 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.490+0000 7faa99fef700 1 --2- 172.21.15.88:0/2449942208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94105cf0 0x7faa941080d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faa840098a0 tx=0x7faa84009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:16.144 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.491+0000 7faa99fef700 1 -- 172.21.15.88:0/2449942208 shutdown_connections 2023-07-14T17:36:16.144 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.491+0000 7faa99fef700 1 --2- 172.21.15.88:0/2449942208 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94105cf0 0x7faa941080d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:16.144 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.491+0000 7faa99fef700 1 -- 172.21.15.88:0/2449942208 >> 172.21.15.88:0/2449942208 conn(0x7faa940ff770 msgr2=0x7faa94101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:16.145 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.491+0000 7faa99fef700 1 -- 172.21.15.88:0/2449942208 shutdown_connections 2023-07-14T17:36:16.145 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.491+0000 7faa99fef700 1 -- 172.21.15.88:0/2449942208 wait complete. 2023-07-14T17:36:16.145 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.491+0000 7faa99fef700 1 Processor -- start 2023-07-14T17:36:16.145 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.491+0000 7faa99fef700 1 -- start start 2023-07-14T17:36:16.146 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.492+0000 7faa99fef700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94105cf0 0x7faa94103ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:16.146 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.492+0000 7faa99fef700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa94105830 con 0x7faa94105cf0 2023-07-14T17:36:16.146 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.492+0000 7faa937fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94105cf0 0x7faa94103ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:16.147 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.492+0000 7faa937fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94105cf0 0x7faa94103ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40184/0 (socket says 172.21.15.88:40184) 2023-07-14T17:36:16.147 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.492+0000 7faa937fe700 1 -- 172.21.15.88:0/216834171 learned_addr learned my addr 172.21.15.88:0/216834171 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:16.147 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.492+0000 7faa937fe700 1 -- 172.21.15.88:0/216834171 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa84009550 con 0x7faa94105cf0 2023-07-14T17:36:16.147 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.492+0000 7faa937fe700 1 --2- 172.21.15.88:0/216834171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94105cf0 0x7faa94103ca0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7faa84006af0 tx=0x7faa84011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:16.148 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.492+0000 7faa90ff9700 1 -- 172.21.15.88:0/216834171 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa84003bf0 con 0x7faa94105cf0 2023-07-14T17:36:16.148 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.492+0000 7faa90ff9700 1 -- 172.21.15.88:0/216834171 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7faa8401b8b0 con 0x7faa94105cf0 2023-07-14T17:36:16.148 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.492+0000 7faa90ff9700 1 -- 172.21.15.88:0/216834171 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faa840198b0 con 0x7faa94105cf0 2023-07-14T17:36:16.149 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.492+0000 7faa99fef700 1 -- 172.21.15.88:0/216834171 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa94105a30 con 0x7faa94105cf0 2023-07-14T17:36:16.149 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.492+0000 7faa99fef700 1 -- 172.21.15.88:0/216834171 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa941043b0 con 0x7faa94105cf0 2023-07-14T17:36:16.149 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.493+0000 7faa90ff9700 1 -- 172.21.15.88:0/216834171 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7faa84019b70 con 0x7faa94105cf0 2023-07-14T17:36:16.149 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.493+0000 7faa90ff9700 1 --2- 172.21.15.88:0/216834171 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7faa7c0386c0 0x7faa7c03ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:16.150 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.493+0000 7faa90ff9700 1 -- 172.21.15.88:0/216834171 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7faa8404ca80 con 0x7faa94105cf0 2023-07-14T17:36:16.150 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.493+0000 7faa92ffd700 1 -- 172.21.15.88:0/216834171 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7faa7c0386c0 msgr2=0x7faa7c03ab70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.88:6800/1369181448 2023-07-14T17:36:16.150 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.493+0000 7faa92ffd700 1 --2- 172.21.15.88:0/216834171 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7faa7c0386c0 0x7faa7c03ab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T17:36:16.150 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.494+0000 7faa99fef700 1 -- 172.21.15.88:0/216834171 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa94061c70 con 0x7faa94105cf0 2023-07-14T17:36:16.151 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.498+0000 7faa90ff9700 1 -- 172.21.15.88:0/216834171 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faa84010380 con 0x7faa94105cf0 2023-07-14T17:36:16.151 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.694+0000 7faa92ffd700 1 -- 172.21.15.88:0/216834171 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7faa7c0386c0 msgr2=0x7faa7c03ab70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.88:6800/1369181448 2023-07-14T17:36:16.151 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.694+0000 7faa92ffd700 1 --2- 172.21.15.88:0/216834171 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7faa7c0386c0 0x7faa7c03ab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T17:36:16.152 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.745+0000 7faa99fef700 1 -- 172.21.15.88:0/216834171 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7faa94104af0 con 0x7faa94105cf0 2023-07-14T17:36:16.152 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.745+0000 7faa90ff9700 1 -- 172.21.15.88:0/216834171 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7faa84018b40 con 0x7faa94105cf0 2023-07-14T17:36:16.152 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.747+0000 7faa99fef700 1 -- 172.21.15.88:0/216834171 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7faa7c0386c0 msgr2=0x7faa7c03ab70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T17:36:16.152 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.747+0000 7faa99fef700 1 --2- 172.21.15.88:0/216834171 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7faa7c0386c0 0x7faa7c03ab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:16.153 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.747+0000 7faa99fef700 1 -- 172.21.15.88:0/216834171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94105cf0 msgr2=0x7faa94103ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:16.153 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.747+0000 7faa99fef700 1 --2- 172.21.15.88:0/216834171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94105cf0 0x7faa94103ca0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7faa84006af0 tx=0x7faa84011a30 comp rx=0 tx=0).stop 2023-07-14T17:36:16.153 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.747+0000 7faa99fef700 1 -- 172.21.15.88:0/216834171 shutdown_connections 2023-07-14T17:36:16.153 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.747+0000 7faa99fef700 1 --2- 172.21.15.88:0/216834171 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7faa7c0386c0 0x7faa7c03ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:16.154 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.747+0000 7faa99fef700 1 --2- 172.21.15.88:0/216834171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faa94105cf0 0x7faa94103ca0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:16.154 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.747+0000 7faa99fef700 1 -- 172.21.15.88:0/216834171 >> 172.21.15.88:0/216834171 conn(0x7faa940ff770 msgr2=0x7faa94100510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:16.154 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.747+0000 7faa99fef700 1 -- 172.21.15.88:0/216834171 shutdown_connections 2023-07-14T17:36:16.154 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:15.747+0000 7faa99fef700 1 -- 172.21.15.88:0/216834171 wait complete. 2023-07-14T17:36:16.155 INFO:teuthology.orchestra.run.smithi088.stdout:Waiting for the mgr to restart... 2023-07-14T17:36:16.155 INFO:teuthology.orchestra.run.smithi088.stdout:Waiting for mgr epoch 5... 2023-07-14T17:36:16.600 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:16 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:16.305+0000 7f757aaae200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T17:36:16.870 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:16 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/216834171' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-14T17:36:17.196 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:16 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:16.883+0000 7f757aaae200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T17:36:17.196 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:17 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:17.050+0000 7f757aaae200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T17:36:18.049 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:17 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:17.805+0000 7f757aaae200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T17:36:18.049 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:17 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:17.894+0000 7f757aaae200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T17:36:18.321 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:18 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:18.062+0000 7f757aaae200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T17:36:18.690 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:18 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:18.335+0000 7f757aaae200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T17:36:18.690 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:18 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:18.559+0000 7f757aaae200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T17:36:18.946 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:18 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:18.703+0000 7f757aaae200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T17:36:19.446 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:18 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:18.960+0000 7f757aaae200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T17:36:19.946 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:19 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:19.650+0000 7f757aaae200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T17:36:19.946 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:19 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:19.777+0000 7f757aaae200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T17:36:20.273 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:19 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:19.963+0000 7f757aaae200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T17:36:20.274 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:20 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:20.075+0000 7f757aaae200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T17:36:20.696 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:20 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:20.286+0000 7f757aaae200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T17:36:21.446 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:21.140+0000 7f757aaae200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T17:36:21.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: Active manager daemon x restarted 2023-07-14T17:36:21.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: Activating manager daemon x 2023-07-14T17:36:21.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: osdmap e2: 0 total, 0 up, 0 in 2023-07-14T17:36:21.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: mgrmap e6: x(active, starting, since 0.00319158s) 2023-07-14T17:36:21.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T17:36:21.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T17:36:21.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T17:36:21.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T17:36:21.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T17:36:21.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: Manager daemon x is now available 2023-07-14T17:36:21.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' 2023-07-14T17:36:21.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' 2023-07-14T17:36:21.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:36:21.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:36:21.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:36:21.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T17:36:21.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T17:36:21.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' 2023-07-14T17:36:21.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:21 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' 2023-07-14T17:36:22.596 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout { 2023-07-14T17:36:22.596 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-07-14T17:36:22.596 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-14T17:36:22.596 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout } 2023-07-14T17:36:22.597 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.581+0000 7f2069115700 1 Processor -- start 2023-07-14T17:36:22.597 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.581+0000 7f2069115700 1 -- start start 2023-07-14T17:36:22.597 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.581+0000 7f2069115700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2064105d30 0x7f2064108110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:22.597 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.581+0000 7f2069115700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2064079d10 con 0x7f2064105d30 2023-07-14T17:36:22.597 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.582+0000 7f2062d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2064105d30 0x7f2064108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:22.597 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.582+0000 7f2062d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2064105d30 0x7f2064108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40192/0 (socket says 172.21.15.88:40192) 2023-07-14T17:36:22.597 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.582+0000 7f2062d9d700 1 -- 172.21.15.88:0/3869715197 learned_addr learned my addr 172.21.15.88:0/3869715197 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:22.598 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.582+0000 7f2062d9d700 1 -- 172.21.15.88:0/3869715197 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2064079e50 con 0x7f2064105d30 2023-07-14T17:36:22.598 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.582+0000 7f2062d9d700 1 --2- 172.21.15.88:0/3869715197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2064105d30 0x7f2064108110 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f20540098a0 tx=0x7f2054009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba3c42e32ba88c7c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:22.598 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.582+0000 7f2061d9b700 1 -- 172.21.15.88:0/3869715197 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2054004160 con 0x7f2064105d30 2023-07-14T17:36:22.598 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.582+0000 7f2061d9b700 1 -- 172.21.15.88:0/3869715197 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f20540042c0 con 0x7f2064105d30 2023-07-14T17:36:22.598 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.582+0000 7f2061d9b700 1 -- 172.21.15.88:0/3869715197 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2054011940 con 0x7f2064105d30 2023-07-14T17:36:22.598 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.583+0000 7f2069115700 1 -- 172.21.15.88:0/3869715197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2064105d30 msgr2=0x7f2064108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:22.598 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.583+0000 7f2069115700 1 --2- 172.21.15.88:0/3869715197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2064105d30 0x7f2064108110 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f20540098a0 tx=0x7f2054009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:22.599 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.583+0000 7f2069115700 1 -- 172.21.15.88:0/3869715197 shutdown_connections 2023-07-14T17:36:22.599 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.583+0000 7f2069115700 1 --2- 172.21.15.88:0/3869715197 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2064105d30 0x7f2064108110 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:22.599 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.583+0000 7f2069115700 1 -- 172.21.15.88:0/3869715197 >> 172.21.15.88:0/3869715197 conn(0x7f20640ff770 msgr2=0x7f2064101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:22.599 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.583+0000 7f2069115700 1 -- 172.21.15.88:0/3869715197 shutdown_connections 2023-07-14T17:36:22.599 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.583+0000 7f2069115700 1 -- 172.21.15.88:0/3869715197 wait complete. 2023-07-14T17:36:22.599 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.584+0000 7f2069115700 1 Processor -- start 2023-07-14T17:36:22.600 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.584+0000 7f2069115700 1 -- start start 2023-07-14T17:36:22.600 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.584+0000 7f2069115700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2064105d30 0x7f2064103ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:22.600 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.584+0000 7f2069115700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2064105870 con 0x7f2064105d30 2023-07-14T17:36:22.600 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.584+0000 7f2062d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2064105d30 0x7f2064103ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:22.600 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.584+0000 7f2062d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2064105d30 0x7f2064103ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40194/0 (socket says 172.21.15.88:40194) 2023-07-14T17:36:22.600 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.584+0000 7f2062d9d700 1 -- 172.21.15.88:0/402350176 learned_addr learned my addr 172.21.15.88:0/402350176 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:22.600 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.585+0000 7f2062d9d700 1 -- 172.21.15.88:0/402350176 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2054009550 con 0x7f2064105d30 2023-07-14T17:36:22.601 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.585+0000 7f2062d9d700 1 --2- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2064105d30 0x7f2064103ce0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2054000c00 tx=0x7f2054011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:22.601 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.585+0000 7f204bfff700 1 -- 172.21.15.88:0/402350176 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f20540036a0 con 0x7f2064105d30 2023-07-14T17:36:22.601 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.585+0000 7f204bfff700 1 -- 172.21.15.88:0/402350176 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f205401b440 con 0x7f2064105d30 2023-07-14T17:36:22.601 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.585+0000 7f2069115700 1 -- 172.21.15.88:0/402350176 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2064105a70 con 0x7f2064105d30 2023-07-14T17:36:22.601 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.585+0000 7f204bfff700 1 -- 172.21.15.88:0/402350176 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f20540193f0 con 0x7f2064105d30 2023-07-14T17:36:22.601 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.585+0000 7f2069115700 1 -- 172.21.15.88:0/402350176 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20641043f0 con 0x7f2064105d30 2023-07-14T17:36:22.601 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.586+0000 7f204bfff700 1 -- 172.21.15.88:0/402350176 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f2054019620 con 0x7f2064105d30 2023-07-14T17:36:22.602 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.586+0000 7f204bfff700 1 --2- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f204c038760 0x7f204c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:22.602 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.586+0000 7f204bfff700 1 -- 172.21.15.88:0/402350176 --> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f204c03b320 con 0x7f204c038760 2023-07-14T17:36:22.602 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.586+0000 7f206259c700 1 -- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f204c038760 msgr2=0x7f204c03ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.88:6800/1369181448 2023-07-14T17:36:22.602 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.586+0000 7f204bfff700 1 -- 172.21.15.88:0/402350176 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f205404c5b0 con 0x7f2064105d30 2023-07-14T17:36:22.602 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.586+0000 7f206259c700 1 --2- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f204c038760 0x7f204c03ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T17:36:22.602 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.787+0000 7f206259c700 1 -- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f204c038760 msgr2=0x7f204c03ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.88:6800/1369181448 2023-07-14T17:36:22.602 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:16.787+0000 7f206259c700 1 --2- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f204c038760 0x7f204c03ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T17:36:22.602 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:17.188+0000 7f206259c700 1 -- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f204c038760 msgr2=0x7f204c03ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.88:6800/1369181448 2023-07-14T17:36:22.603 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:17.188+0000 7f206259c700 1 --2- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f204c038760 0x7f204c03ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-14T17:36:22.603 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:17.989+0000 7f206259c700 1 -- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f204c038760 msgr2=0x7f204c03ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.88:6800/1369181448 2023-07-14T17:36:22.603 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:17.989+0000 7f206259c700 1 --2- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f204c038760 0x7f204c03ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-14T17:36:22.603 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:19.590+0000 7f206259c700 1 -- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f204c038760 msgr2=0x7f204c03ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.88:6800/1369181448 2023-07-14T17:36:22.603 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:19.590+0000 7f206259c700 1 --2- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f204c038760 0x7f204c03ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-14T17:36:22.603 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:21.155+0000 7f204bfff700 1 -- 172.21.15.88:0/402350176 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7f2054018c70 con 0x7f2064105d30 2023-07-14T17:36:22.603 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:21.155+0000 7f204bfff700 1 -- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f204c038760 msgr2=0x7f204c03ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T17:36:22.604 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:21.155+0000 7f204bfff700 1 --2- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f204c038760 0x7f204c03ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:22.604 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.156+0000 7f204bfff700 1 -- 172.21.15.88:0/402350176 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7f205401b5b0 con 0x7f2064105d30 2023-07-14T17:36:22.604 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.156+0000 7f204bfff700 1 --2- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f204c03bfe0 0x7f204c03e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:22.604 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.156+0000 7f204bfff700 1 -- 172.21.15.88:0/402350176 --> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f204c03b320 con 0x7f204c03bfe0 2023-07-14T17:36:22.604 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.156+0000 7f206259c700 1 --2- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f204c03bfe0 0x7f204c03e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:22.604 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.157+0000 7f206259c700 1 --2- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f204c03bfe0 0x7f204c03e3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2058003a10 tx=0x7f20580092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:22.604 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.157+0000 7f204bfff700 1 -- 172.21.15.88:0/402350176 <== mgr.14120 v2:172.21.15.88:6800/2424184187 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f204c03b320 con 0x7f204c03bfe0 2023-07-14T17:36:22.605 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.163+0000 7f2069115700 1 -- 172.21.15.88:0/402350176 --> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f2064105230 con 0x7f204c03bfe0 2023-07-14T17:36:22.605 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.164+0000 7f204bfff700 1 -- 172.21.15.88:0/402350176 <== mgr.14120 v2:172.21.15.88:6800/2424184187 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f2064105230 con 0x7f204c03bfe0 2023-07-14T17:36:22.605 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.164+0000 7f2069115700 1 -- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f204c03bfe0 msgr2=0x7f204c03e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:22.605 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.164+0000 7f2069115700 1 --2- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f204c03bfe0 0x7f204c03e3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2058003a10 tx=0x7f20580092b0 comp rx=0 tx=0).stop 2023-07-14T17:36:22.605 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.164+0000 7f2069115700 1 -- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2064105d30 msgr2=0x7f2064103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:22.605 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.164+0000 7f2069115700 1 --2- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2064105d30 0x7f2064103ce0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2054000c00 tx=0x7f2054011e70 comp rx=0 tx=0).stop 2023-07-14T17:36:22.605 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.164+0000 7f2069115700 1 -- 172.21.15.88:0/402350176 shutdown_connections 2023-07-14T17:36:22.606 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.164+0000 7f2069115700 1 --2- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f204c03bfe0 0x7f204c03e3c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:22.606 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.164+0000 7f2069115700 1 --2- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2064105d30 0x7f2064103ce0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:22.606 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.164+0000 7f2069115700 1 --2- 172.21.15.88:0/402350176 >> [v2:172.21.15.88:6800/1369181448,v1:172.21.15.88:6801/1369181448] conn(0x7f204c038760 0x7f204c03ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:22.606 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.164+0000 7f2069115700 1 -- 172.21.15.88:0/402350176 >> 172.21.15.88:0/402350176 conn(0x7f20640ff770 msgr2=0x7f2064100510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:22.606 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.164+0000 7f2069115700 1 -- 172.21.15.88:0/402350176 shutdown_connections 2023-07-14T17:36:22.606 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:22.164+0000 7f2069115700 1 -- 172.21.15.88:0/402350176 wait complete. 2023-07-14T17:36:22.606 INFO:teuthology.orchestra.run.smithi088.stdout:mgr epoch 5 is available 2023-07-14T17:36:22.607 INFO:teuthology.orchestra.run.smithi088.stdout:Setting orchestrator backend to cephadm... 2023-07-14T17:36:22.796 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:22 smithi088 ceph-mon[127465]: Found migration_current of "None". Setting to last migration. 2023-07-14T17:36:22.797 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:22 smithi088 ceph-mon[127465]: mgrmap e7: x(active, since 1.00534s) 2023-07-14T17:36:22.797 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:22 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:36:23.639 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.057+0000 7fadb14f4700 1 Processor -- start 2023-07-14T17:36:23.639 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.057+0000 7fadb14f4700 1 -- start start 2023-07-14T17:36:23.639 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.058+0000 7fadb14f4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadac105d30 0x7fadac108110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:23.640 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.058+0000 7fadb14f4700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadac079d10 con 0x7fadac105d30 2023-07-14T17:36:23.640 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.058+0000 7fadaaffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadac105d30 0x7fadac108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:23.640 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.058+0000 7fadaaffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadac105d30 0x7fadac108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40246/0 (socket says 172.21.15.88:40246) 2023-07-14T17:36:23.640 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.058+0000 7fadaaffd700 1 -- 172.21.15.88:0/329979054 learned_addr learned my addr 172.21.15.88:0/329979054 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:23.641 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.058+0000 7fadaaffd700 1 -- 172.21.15.88:0/329979054 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadac079e50 con 0x7fadac105d30 2023-07-14T17:36:23.641 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.058+0000 7fadaaffd700 1 --2- 172.21.15.88:0/329979054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadac105d30 0x7fadac108110 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fada00098a0 tx=0x7fada0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e6f93e164ed848e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:23.641 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.058+0000 7fada9ffb700 1 -- 172.21.15.88:0/329979054 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fada0004160 con 0x7fadac105d30 2023-07-14T17:36:23.641 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.058+0000 7fada9ffb700 1 -- 172.21.15.88:0/329979054 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fada00042c0 con 0x7fadac105d30 2023-07-14T17:36:23.641 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.058+0000 7fada9ffb700 1 -- 172.21.15.88:0/329979054 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fada0011940 con 0x7fadac105d30 2023-07-14T17:36:23.642 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.059+0000 7fadb14f4700 1 -- 172.21.15.88:0/329979054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadac105d30 msgr2=0x7fadac108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:23.642 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.059+0000 7fadb14f4700 1 --2- 172.21.15.88:0/329979054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadac105d30 0x7fadac108110 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fada00098a0 tx=0x7fada0009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:23.642 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.059+0000 7fadb14f4700 1 -- 172.21.15.88:0/329979054 shutdown_connections 2023-07-14T17:36:23.642 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.059+0000 7fadb14f4700 1 --2- 172.21.15.88:0/329979054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadac105d30 0x7fadac108110 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:23.643 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.059+0000 7fadb14f4700 1 -- 172.21.15.88:0/329979054 >> 172.21.15.88:0/329979054 conn(0x7fadac0ff750 msgr2=0x7fadac101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:23.643 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.059+0000 7fadb14f4700 1 -- 172.21.15.88:0/329979054 shutdown_connections 2023-07-14T17:36:23.643 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.059+0000 7fadb14f4700 1 -- 172.21.15.88:0/329979054 wait complete. 2023-07-14T17:36:23.643 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.059+0000 7fadb14f4700 1 Processor -- start 2023-07-14T17:36:23.643 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.060+0000 7fadb14f4700 1 -- start start 2023-07-14T17:36:23.644 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.060+0000 7fadb14f4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadac105d30 0x7fadac198e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:23.644 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.060+0000 7fadb14f4700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadac1993c0 con 0x7fadac105d30 2023-07-14T17:36:23.644 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.060+0000 7fadaaffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadac105d30 0x7fadac198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:23.644 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.060+0000 7fadaaffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadac105d30 0x7fadac198e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40248/0 (socket says 172.21.15.88:40248) 2023-07-14T17:36:23.644 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.060+0000 7fadaaffd700 1 -- 172.21.15.88:0/1416645459 learned_addr learned my addr 172.21.15.88:0/1416645459 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:23.645 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.060+0000 7fadaaffd700 1 -- 172.21.15.88:0/1416645459 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada0009550 con 0x7fadac105d30 2023-07-14T17:36:23.645 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.060+0000 7fadaaffd700 1 --2- 172.21.15.88:0/1416645459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadac105d30 0x7fadac198e80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fada0000c00 tx=0x7fada0011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:23.645 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.060+0000 7fad93fff700 1 -- 172.21.15.88:0/1416645459 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fada00036a0 con 0x7fadac105d30 2023-07-14T17:36:23.645 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.060+0000 7fadb14f4700 1 -- 172.21.15.88:0/1416645459 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadac1995c0 con 0x7fadac105d30 2023-07-14T17:36:23.645 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.060+0000 7fad93fff700 1 -- 172.21.15.88:0/1416645459 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fada001b440 con 0x7fadac105d30 2023-07-14T17:36:23.646 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.060+0000 7fad93fff700 1 -- 172.21.15.88:0/1416645459 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fada00193f0 con 0x7fadac105d30 2023-07-14T17:36:23.646 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.061+0000 7fadb14f4700 1 -- 172.21.15.88:0/1416645459 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadac19a250 con 0x7fadac105d30 2023-07-14T17:36:23.646 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.061+0000 7fad93fff700 1 -- 172.21.15.88:0/1416645459 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fada0003800 con 0x7fadac105d30 2023-07-14T17:36:23.646 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.061+0000 7fad93fff700 1 --2- 172.21.15.88:0/1416645459 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fad940385e0 0x7fad9403aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:23.646 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.061+0000 7fadb14f4700 1 -- 172.21.15.88:0/1416645459 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadac061c70 con 0x7fadac105d30 2023-07-14T17:36:23.647 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.062+0000 7fad93fff700 1 -- 172.21.15.88:0/1416645459 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fada004c440 con 0x7fadac105d30 2023-07-14T17:36:23.647 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.062+0000 7fadaa7fc700 1 --2- 172.21.15.88:0/1416645459 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fad940385e0 0x7fad9403aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:23.647 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.062+0000 7fadaa7fc700 1 --2- 172.21.15.88:0/1416645459 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fad940385e0 0x7fad9403aa90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fad9c006fd0 tx=0x7fad9c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:23.647 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.066+0000 7fad93fff700 1 -- 172.21.15.88:0/1416645459 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fada001b940 con 0x7fadac105d30 2023-07-14T17:36:23.648 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.263+0000 7fadb14f4700 1 -- 172.21.15.88:0/1416645459 --> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fadac199a40 con 0x7fad940385e0 2023-07-14T17:36:23.648 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.268+0000 7fad93fff700 1 -- 172.21.15.88:0/1416645459 <== mgr.14120 v2:172.21.15.88:6800/2424184187 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fadac199a40 con 0x7fad940385e0 2023-07-14T17:36:23.648 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.270+0000 7fadb14f4700 1 -- 172.21.15.88:0/1416645459 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fad940385e0 msgr2=0x7fad9403aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:23.648 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.270+0000 7fadb14f4700 1 --2- 172.21.15.88:0/1416645459 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fad940385e0 0x7fad9403aa90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fad9c006fd0 tx=0x7fad9c006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:23.648 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.271+0000 7fadb14f4700 1 -- 172.21.15.88:0/1416645459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadac105d30 msgr2=0x7fadac198e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:23.649 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.271+0000 7fadb14f4700 1 --2- 172.21.15.88:0/1416645459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadac105d30 0x7fadac198e80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fada0000c00 tx=0x7fada0011e70 comp rx=0 tx=0).stop 2023-07-14T17:36:23.649 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.271+0000 7fadb14f4700 1 -- 172.21.15.88:0/1416645459 shutdown_connections 2023-07-14T17:36:23.649 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.271+0000 7fadb14f4700 1 --2- 172.21.15.88:0/1416645459 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fad940385e0 0x7fad9403aa90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:23.649 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.271+0000 7fadb14f4700 1 --2- 172.21.15.88:0/1416645459 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fadac105d30 0x7fadac198e80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:23.649 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.271+0000 7fadb14f4700 1 -- 172.21.15.88:0/1416645459 >> 172.21.15.88:0/1416645459 conn(0x7fadac0ff750 msgr2=0x7fadac1004e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:23.650 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.271+0000 7fadb14f4700 1 -- 172.21.15.88:0/1416645459 shutdown_connections 2023-07-14T17:36:23.650 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:23.271+0000 7fadb14f4700 1 -- 172.21.15.88:0/1416645459 wait complete. 2023-07-14T17:36:23.854 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:23 smithi088 ceph-mon[127465]: [14/Jul/2023:17:36:21] ENGINE Bus STARTING 2023-07-14T17:36:23.854 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:23 smithi088 ceph-mon[127465]: [14/Jul/2023:17:36:22] ENGINE Serving on http://172.21.15.88:8765 2023-07-14T17:36:23.854 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:23 smithi088 ceph-mon[127465]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-14T17:36:23.855 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:23 smithi088 ceph-mon[127465]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-14T17:36:23.855 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:23 smithi088 ceph-mon[127465]: [14/Jul/2023:17:36:22] ENGINE Serving on https://172.21.15.88:7150 2023-07-14T17:36:23.855 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:23 smithi088 ceph-mon[127465]: [14/Jul/2023:17:36:22] ENGINE Bus STARTED 2023-07-14T17:36:23.855 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:23 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' 2023-07-14T17:36:23.855 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:23 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:36:24.679 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout value unchanged 2023-07-14T17:36:24.679 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.089+0000 7ff2386b8700 1 Processor -- start 2023-07-14T17:36:24.679 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.089+0000 7ff2386b8700 1 -- start start 2023-07-14T17:36:24.679 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.090+0000 7ff2386b8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff230103b30 0x7ff230103f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:24.679 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.090+0000 7ff2386b8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff230079d10 con 0x7ff230103b30 2023-07-14T17:36:24.680 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.090+0000 7ff236454700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff230103b30 0x7ff230103f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:24.680 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.090+0000 7ff236454700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff230103b30 0x7ff230103f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40252/0 (socket says 172.21.15.88:40252) 2023-07-14T17:36:24.680 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.090+0000 7ff236454700 1 -- 172.21.15.88:0/1627947726 learned_addr learned my addr 172.21.15.88:0/1627947726 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:24.681 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.090+0000 7ff236454700 1 -- 172.21.15.88:0/1627947726 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff230079e50 con 0x7ff230103b30 2023-07-14T17:36:24.681 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.090+0000 7ff236454700 1 --2- 172.21.15.88:0/1627947726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff230103b30 0x7ff230103f40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff2200098a0 tx=0x7ff220009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a27ea351c6309ded server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:24.681 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.090+0000 7ff235452700 1 -- 172.21.15.88:0/1627947726 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff220004160 con 0x7ff230103b30 2023-07-14T17:36:24.681 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.090+0000 7ff235452700 1 -- 172.21.15.88:0/1627947726 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7ff2200042c0 con 0x7ff230103b30 2023-07-14T17:36:24.681 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.090+0000 7ff235452700 1 -- 172.21.15.88:0/1627947726 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff220011940 con 0x7ff230103b30 2023-07-14T17:36:24.681 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.091+0000 7ff2386b8700 1 -- 172.21.15.88:0/1627947726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff230103b30 msgr2=0x7ff230103f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:24.681 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.091+0000 7ff2386b8700 1 --2- 172.21.15.88:0/1627947726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff230103b30 0x7ff230103f40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff2200098a0 tx=0x7ff220009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:24.682 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.091+0000 7ff2386b8700 1 -- 172.21.15.88:0/1627947726 shutdown_connections 2023-07-14T17:36:24.682 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.091+0000 7ff2386b8700 1 --2- 172.21.15.88:0/1627947726 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff230103b30 0x7ff230103f40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:24.682 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.091+0000 7ff2386b8700 1 -- 172.21.15.88:0/1627947726 >> 172.21.15.88:0/1627947726 conn(0x7ff2300ff7a0 msgr2=0x7ff230101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:24.682 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.091+0000 7ff2386b8700 1 -- 172.21.15.88:0/1627947726 shutdown_connections 2023-07-14T17:36:24.682 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.091+0000 7ff2386b8700 1 -- 172.21.15.88:0/1627947726 wait complete. 2023-07-14T17:36:24.682 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.092+0000 7ff2386b8700 1 Processor -- start 2023-07-14T17:36:24.682 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.092+0000 7ff2386b8700 1 -- start start 2023-07-14T17:36:24.683 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.092+0000 7ff2386b8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff230103b30 0x7ff23019b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:24.683 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.092+0000 7ff2386b8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23019b6b0 con 0x7ff230103b30 2023-07-14T17:36:24.683 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.092+0000 7ff236454700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff230103b30 0x7ff23019b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:24.683 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.092+0000 7ff236454700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff230103b30 0x7ff23019b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40254/0 (socket says 172.21.15.88:40254) 2023-07-14T17:36:24.683 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.092+0000 7ff236454700 1 -- 172.21.15.88:0/2520505186 learned_addr learned my addr 172.21.15.88:0/2520505186 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:24.683 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.092+0000 7ff236454700 1 -- 172.21.15.88:0/2520505186 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff220009550 con 0x7ff230103b30 2023-07-14T17:36:24.683 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.092+0000 7ff236454700 1 --2- 172.21.15.88:0/2520505186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff230103b30 0x7ff23019b170 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff220006af0 tx=0x7ff220011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:24.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.092+0000 7ff2277fe700 1 -- 172.21.15.88:0/2520505186 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff220003bf0 con 0x7ff230103b30 2023-07-14T17:36:24.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.092+0000 7ff2277fe700 1 -- 172.21.15.88:0/2520505186 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7ff22001b8b0 con 0x7ff230103b30 2023-07-14T17:36:24.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.092+0000 7ff2386b8700 1 -- 172.21.15.88:0/2520505186 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff23019b910 con 0x7ff230103b30 2023-07-14T17:36:24.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.093+0000 7ff2277fe700 1 -- 172.21.15.88:0/2520505186 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff2200198b0 con 0x7ff230103b30 2023-07-14T17:36:24.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.093+0000 7ff2386b8700 1 -- 172.21.15.88:0/2520505186 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff23019c540 con 0x7ff230103b30 2023-07-14T17:36:24.684 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.093+0000 7ff2277fe700 1 -- 172.21.15.88:0/2520505186 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7ff220019b70 con 0x7ff230103b30 2023-07-14T17:36:24.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.093+0000 7ff2386b8700 1 -- 172.21.15.88:0/2520505186 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff230061c70 con 0x7ff230103b30 2023-07-14T17:36:24.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.093+0000 7ff2277fe700 1 --2- 172.21.15.88:0/2520505186 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7ff21c0386c0 0x7ff21c03ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:24.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.094+0000 7ff2277fe700 1 -- 172.21.15.88:0/2520505186 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff22004cc50 con 0x7ff230103b30 2023-07-14T17:36:24.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.094+0000 7ff235c53700 1 --2- 172.21.15.88:0/2520505186 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7ff21c0386c0 0x7ff21c03ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:24.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.094+0000 7ff235c53700 1 --2- 172.21.15.88:0/2520505186 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7ff21c0386c0 0x7ff21c03ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff22c006fd0 tx=0x7ff22c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:24.685 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.098+0000 7ff2277fe700 1 -- 172.21.15.88:0/2520505186 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff220052020 con 0x7ff230103b30 2023-07-14T17:36:24.686 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.282+0000 7ff2386b8700 1 -- 172.21.15.88:0/2520505186 --> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7ff23019c780 con 0x7ff21c0386c0 2023-07-14T17:36:24.686 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.283+0000 7ff2277fe700 1 -- 172.21.15.88:0/2520505186 <== mgr.14120 v2:172.21.15.88:6800/2424184187 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7ff23019c780 con 0x7ff21c0386c0 2023-07-14T17:36:24.686 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.285+0000 7ff2386b8700 1 -- 172.21.15.88:0/2520505186 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7ff21c0386c0 msgr2=0x7ff21c03ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:24.686 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.285+0000 7ff2386b8700 1 --2- 172.21.15.88:0/2520505186 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7ff21c0386c0 0x7ff21c03ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff22c006fd0 tx=0x7ff22c006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:24.686 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.285+0000 7ff2386b8700 1 -- 172.21.15.88:0/2520505186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff230103b30 msgr2=0x7ff23019b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:24.686 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.285+0000 7ff2386b8700 1 --2- 172.21.15.88:0/2520505186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff230103b30 0x7ff23019b170 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff220006af0 tx=0x7ff220011a30 comp rx=0 tx=0).stop 2023-07-14T17:36:24.686 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.285+0000 7ff2386b8700 1 -- 172.21.15.88:0/2520505186 shutdown_connections 2023-07-14T17:36:24.687 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.285+0000 7ff2386b8700 1 --2- 172.21.15.88:0/2520505186 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7ff21c0386c0 0x7ff21c03ab70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:24.687 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.285+0000 7ff2386b8700 1 --2- 172.21.15.88:0/2520505186 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff230103b30 0x7ff23019b170 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:24.687 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.285+0000 7ff2386b8700 1 -- 172.21.15.88:0/2520505186 >> 172.21.15.88:0/2520505186 conn(0x7ff2300ff7a0 msgr2=0x7ff23018f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:24.687 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.285+0000 7ff2386b8700 1 -- 172.21.15.88:0/2520505186 shutdown_connections 2023-07-14T17:36:24.687 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:24.285+0000 7ff2386b8700 1 -- 172.21.15.88:0/2520505186 wait complete. 2023-07-14T17:36:24.687 INFO:teuthology.orchestra.run.smithi088.stdout:Generating ssh key... 2023-07-14T17:36:24.870 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:24 smithi088 ceph-mon[127465]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:36:24.870 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:24 smithi088 ceph-mon[127465]: mgrmap e8: x(active, since 2s) 2023-07-14T17:36:25.923 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.172+0000 7f5fb39e9700 1 Processor -- start 2023-07-14T17:36:25.923 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.172+0000 7f5fb39e9700 1 -- start start 2023-07-14T17:36:25.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.172+0000 7f5fb39e9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fac079cd0 0x7f5fac10a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:25.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.172+0000 7f5fb39e9700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac105b80 con 0x7f5fac079cd0 2023-07-14T17:36:25.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.172+0000 7f5fb1785700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fac079cd0 0x7f5fac10a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:25.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.172+0000 7f5fb1785700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fac079cd0 0x7f5fac10a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40258/0 (socket says 172.21.15.88:40258) 2023-07-14T17:36:25.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.172+0000 7f5fb1785700 1 -- 172.21.15.88:0/2591062836 learned_addr learned my addr 172.21.15.88:0/2591062836 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:25.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.172+0000 7f5fb1785700 1 -- 172.21.15.88:0/2591062836 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fac10a6d0 con 0x7f5fac079cd0 2023-07-14T17:36:25.924 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.172+0000 7f5fb1785700 1 --2- 172.21.15.88:0/2591062836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fac079cd0 0x7f5fac10a190 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c0098a0 tx=0x7f5f9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=300758b4a484b63d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:25.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.173+0000 7f5fa3fff700 1 -- 172.21.15.88:0/2591062836 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f9c004160 con 0x7f5fac079cd0 2023-07-14T17:36:25.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.173+0000 7f5fa3fff700 1 -- 172.21.15.88:0/2591062836 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f5f9c0042c0 con 0x7f5fac079cd0 2023-07-14T17:36:25.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.173+0000 7f5fa3fff700 1 -- 172.21.15.88:0/2591062836 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f9c011940 con 0x7f5fac079cd0 2023-07-14T17:36:25.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.173+0000 7f5fb39e9700 1 -- 172.21.15.88:0/2591062836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fac079cd0 msgr2=0x7f5fac10a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:25.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.173+0000 7f5fb39e9700 1 --2- 172.21.15.88:0/2591062836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fac079cd0 0x7f5fac10a190 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c0098a0 tx=0x7f5f9c009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:25.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.173+0000 7f5fb39e9700 1 -- 172.21.15.88:0/2591062836 shutdown_connections 2023-07-14T17:36:25.925 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.173+0000 7f5fb39e9700 1 --2- 172.21.15.88:0/2591062836 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fac079cd0 0x7f5fac10a190 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:25.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.173+0000 7f5fb39e9700 1 -- 172.21.15.88:0/2591062836 >> 172.21.15.88:0/2591062836 conn(0x7f5fac0ff7a0 msgr2=0x7f5fac101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:25.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.173+0000 7f5fb39e9700 1 -- 172.21.15.88:0/2591062836 shutdown_connections 2023-07-14T17:36:25.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.173+0000 7f5fb39e9700 1 -- 172.21.15.88:0/2591062836 wait complete. 2023-07-14T17:36:25.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.174+0000 7f5fb39e9700 1 Processor -- start 2023-07-14T17:36:25.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.174+0000 7f5fb39e9700 1 -- start start 2023-07-14T17:36:25.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.174+0000 7f5fb39e9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fac19b110 0x7f5fac19b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:25.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.174+0000 7f5fb39e9700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fac19ba60 con 0x7f5fac19b110 2023-07-14T17:36:25.926 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.174+0000 7f5fb1785700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fac19b110 0x7f5fac19b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:25.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.174+0000 7f5fb1785700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fac19b110 0x7f5fac19b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40260/0 (socket says 172.21.15.88:40260) 2023-07-14T17:36:25.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.174+0000 7f5fb1785700 1 -- 172.21.15.88:0/2053193393 learned_addr learned my addr 172.21.15.88:0/2053193393 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:25.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.174+0000 7f5fb1785700 1 -- 172.21.15.88:0/2053193393 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c009550 con 0x7f5fac19b110 2023-07-14T17:36:25.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.175+0000 7f5fb1785700 1 --2- 172.21.15.88:0/2053193393 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fac19b110 0x7f5fac19b520 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c000c00 tx=0x7f5f9c011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:25.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.175+0000 7f5fa27fc700 1 -- 172.21.15.88:0/2053193393 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f9c0036a0 con 0x7f5fac19b110 2023-07-14T17:36:25.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.175+0000 7f5fa27fc700 1 -- 172.21.15.88:0/2053193393 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f5f9c01b440 con 0x7f5fac19b110 2023-07-14T17:36:25.927 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.175+0000 7f5fb39e9700 1 -- 172.21.15.88:0/2053193393 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fac19bc60 con 0x7f5fac19b110 2023-07-14T17:36:25.928 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.175+0000 7f5fa27fc700 1 -- 172.21.15.88:0/2053193393 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f9c0193f0 con 0x7f5fac19b110 2023-07-14T17:36:25.928 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.175+0000 7f5fb39e9700 1 -- 172.21.15.88:0/2053193393 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fac19c6f0 con 0x7f5fac19b110 2023-07-14T17:36:25.928 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.175+0000 7f5fa27fc700 1 -- 172.21.15.88:0/2053193393 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f5f9c003800 con 0x7f5fac19b110 2023-07-14T17:36:25.928 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.176+0000 7f5fb39e9700 1 -- 172.21.15.88:0/2053193393 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fac061c70 con 0x7f5fac19b110 2023-07-14T17:36:25.928 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.176+0000 7f5fa27fc700 1 --2- 172.21.15.88:0/2053193393 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f5f98038760 0x7f5f9803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:25.928 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.176+0000 7f5fa27fc700 1 -- 172.21.15.88:0/2053193393 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5f9c04c570 con 0x7f5fac19b110 2023-07-14T17:36:25.928 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.176+0000 7f5fb0f84700 1 --2- 172.21.15.88:0/2053193393 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f5f98038760 0x7f5f9803ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:25.929 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.176+0000 7f5fb0f84700 1 --2- 172.21.15.88:0/2053193393 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f5f98038760 0x7f5f9803ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8006fd0 tx=0x7f5fa8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:25.929 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.180+0000 7f5fa27fc700 1 -- 172.21.15.88:0/2053193393 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5f9c01b5d0 con 0x7f5fac19b110 2023-07-14T17:36:25.929 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.366+0000 7f5fb39e9700 1 -- 172.21.15.88:0/2053193393 --> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fac100c20 con 0x7f5f98038760 2023-07-14T17:36:25.929 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.618+0000 7f5fa27fc700 1 -- 172.21.15.88:0/2053193393 <== mgr.14120 v2:172.21.15.88:6800/2424184187 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5fac100c20 con 0x7f5f98038760 2023-07-14T17:36:25.929 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.620+0000 7f5fb39e9700 1 -- 172.21.15.88:0/2053193393 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f5f98038760 msgr2=0x7f5f9803ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:25.929 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.620+0000 7f5fb39e9700 1 --2- 172.21.15.88:0/2053193393 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f5f98038760 0x7f5f9803ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8006fd0 tx=0x7f5fa8006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:25.929 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.620+0000 7f5fb39e9700 1 -- 172.21.15.88:0/2053193393 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fac19b110 msgr2=0x7f5fac19b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:25.929 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.620+0000 7f5fb39e9700 1 --2- 172.21.15.88:0/2053193393 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fac19b110 0x7f5fac19b520 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c000c00 tx=0x7f5f9c011e70 comp rx=0 tx=0).stop 2023-07-14T17:36:25.930 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.621+0000 7f5fb39e9700 1 -- 172.21.15.88:0/2053193393 shutdown_connections 2023-07-14T17:36:25.930 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.621+0000 7f5fb39e9700 1 --2- 172.21.15.88:0/2053193393 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f5f98038760 0x7f5f9803ac10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:25.930 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.621+0000 7f5fb39e9700 1 --2- 172.21.15.88:0/2053193393 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5fac19b110 0x7f5fac19b520 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:25.930 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.621+0000 7f5fb39e9700 1 -- 172.21.15.88:0/2053193393 >> 172.21.15.88:0/2053193393 conn(0x7f5fac0ff7a0 msgr2=0x7f5fac100510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:25.930 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.621+0000 7f5fb39e9700 1 -- 172.21.15.88:0/2053193393 shutdown_connections 2023-07-14T17:36:25.930 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:25.621+0000 7f5fb39e9700 1 -- 172.21.15.88:0/2053193393 wait complete. 2023-07-14T17:36:25.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:25 smithi088 ceph-mon[127465]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:36:25.947 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:25 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: Generating public/private rsa key pair. 2023-07-14T17:36:25.947 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:25 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: Your identification has been saved in /tmp/tmp0hq7usny/key. 2023-07-14T17:36:25.947 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:25 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: Your public key has been saved in /tmp/tmp0hq7usny/key.pub. 2023-07-14T17:36:25.947 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:25 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: The key fingerprint is: 2023-07-14T17:36:25.948 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:25 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: SHA256:b8Au/GriQ5IoFwHB9Vsxd/HpAMZmo6j3TdUwoH3H7N0 ceph-b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:36:25.948 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:25 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: The key's randomart image is: 2023-07-14T17:36:25.948 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:25 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: +---[RSA 3072]----+ 2023-07-14T17:36:25.948 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:25 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: |+o.. o.=.o. | 2023-07-14T17:36:25.948 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:25 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: | .. . B=oo+ . | 2023-07-14T17:36:25.948 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:25 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: | . ..o+..o+* | 2023-07-14T17:36:25.948 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:25 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: | . .oo ..=.. .| 2023-07-14T17:36:25.948 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:25 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: | . o.. S . o .E| 2023-07-14T17:36:25.949 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:25 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: |o +..o . + | 2023-07-14T17:36:25.949 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:25 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: |.. o. + + o | 2023-07-14T17:36:25.949 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:25 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: | o .+ o | 2023-07-14T17:36:25.949 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:25 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: | ..+... | 2023-07-14T17:36:25.949 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:25 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: +----[SHA256]-----+ 2023-07-14T17:36:26.901 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:26 smithi088 ceph-mon[127465]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:36:26.901 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:26 smithi088 ceph-mon[127465]: Generating ssh key... 2023-07-14T17:36:26.901 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:26 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' 2023-07-14T17:36:26.902 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:26 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' 2023-07-14T17:36:26.966 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM2T4TXzNTgec4y9vvL8s6pT+SV6AoSJaL6ILEF/+YMDh2EGtE+GNYPLjpg00uX/xx3g5HM9j3rDYngdrGR3Ao8o9BhjVszmZbfCeyKg5ysOmdKKnuSclx3crkfXVMcrNZaHTdlhoF/EaoBwQJYNfn7amGcDiavcE8MX/BLHjw97MUtRVu+nyEesI4AFu0V4vG31XwdCWXmeG2qBbl6KuXZAntns/w5EbhmAaXWPk9gvs8rbeSUp9FZ4SjzZ0cfY2Ho6lWmCVdo0gIOvYi0RUX1AdQ8qisQ71ommeeR7iFg3IhuChlgTIAN7kTDvmajR5cF71+mk8rpOopdc212G6QlYbFgPrUTcUmFYxCEssvmDiplgd5rZ59m15q4mOxbEEN8dkdvpJ0sL57jHylCkuSkrrAJ+FoVOEcj/pyq0tOya86dYwE4TUrYfmblyn6dlllvcVD23Q49E7BSuHr9DTeQlot6iPGg4A7UPPg7JLu7zw7v3m/A2D+MrGQP8szRC8= ceph-b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:36:26.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.367+0000 7fbb595f1700 1 Processor -- start 2023-07-14T17:36:26.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.367+0000 7fbb595f1700 1 -- start start 2023-07-14T17:36:26.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.367+0000 7fbb595f1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb54103860 0x7fbb54103c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:26.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.367+0000 7fbb595f1700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb541041b0 con 0x7fbb54103860 2023-07-14T17:36:26.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.368+0000 7fbb52ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb54103860 0x7fbb54103c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:26.967 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.368+0000 7fbb52ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb54103860 0x7fbb54103c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40264/0 (socket says 172.21.15.88:40264) 2023-07-14T17:36:26.968 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.368+0000 7fbb52ffd700 1 -- 172.21.15.88:0/691877102 learned_addr learned my addr 172.21.15.88:0/691877102 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:26.968 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.368+0000 7fbb52ffd700 1 -- 172.21.15.88:0/691877102 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb541042f0 con 0x7fbb54103860 2023-07-14T17:36:26.968 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.368+0000 7fbb52ffd700 1 --2- 172.21.15.88:0/691877102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb54103860 0x7fbb54103c70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbb440098a0 tx=0x7fbb44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2567dffa77a49794 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:26.968 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.368+0000 7fbb51ffb700 1 -- 172.21.15.88:0/691877102 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb44004160 con 0x7fbb54103860 2023-07-14T17:36:26.968 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.368+0000 7fbb51ffb700 1 -- 172.21.15.88:0/691877102 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fbb440042c0 con 0x7fbb54103860 2023-07-14T17:36:26.968 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.368+0000 7fbb51ffb700 1 -- 172.21.15.88:0/691877102 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb44011940 con 0x7fbb54103860 2023-07-14T17:36:26.968 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.369+0000 7fbb595f1700 1 -- 172.21.15.88:0/691877102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb54103860 msgr2=0x7fbb54103c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:26.969 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.369+0000 7fbb595f1700 1 --2- 172.21.15.88:0/691877102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb54103860 0x7fbb54103c70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbb440098a0 tx=0x7fbb44009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:26.969 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.369+0000 7fbb595f1700 1 -- 172.21.15.88:0/691877102 shutdown_connections 2023-07-14T17:36:26.969 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.369+0000 7fbb595f1700 1 --2- 172.21.15.88:0/691877102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb54103860 0x7fbb54103c70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:26.969 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.369+0000 7fbb595f1700 1 -- 172.21.15.88:0/691877102 >> 172.21.15.88:0/691877102 conn(0x7fbb540feef0 msgr2=0x7fbb54101300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:26.969 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.369+0000 7fbb595f1700 1 -- 172.21.15.88:0/691877102 shutdown_connections 2023-07-14T17:36:26.969 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.369+0000 7fbb595f1700 1 -- 172.21.15.88:0/691877102 wait complete. 2023-07-14T17:36:26.969 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.369+0000 7fbb595f1700 1 Processor -- start 2023-07-14T17:36:26.970 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.370+0000 7fbb595f1700 1 -- start start 2023-07-14T17:36:26.970 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.370+0000 7fbb595f1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb54103860 0x7fbb54198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:26.970 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.370+0000 7fbb595f1700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb54199490 con 0x7fbb54103860 2023-07-14T17:36:26.970 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.370+0000 7fbb52ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb54103860 0x7fbb54198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:26.970 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.370+0000 7fbb52ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb54103860 0x7fbb54198f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40266/0 (socket says 172.21.15.88:40266) 2023-07-14T17:36:26.970 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.370+0000 7fbb52ffd700 1 -- 172.21.15.88:0/1761947091 learned_addr learned my addr 172.21.15.88:0/1761947091 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:26.970 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.370+0000 7fbb52ffd700 1 -- 172.21.15.88:0/1761947091 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb44009550 con 0x7fbb54103860 2023-07-14T17:36:26.970 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.370+0000 7fbb52ffd700 1 --2- 172.21.15.88:0/1761947091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb54103860 0x7fbb54198f50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbb44011dc0 tx=0x7fbb44011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:26.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.370+0000 7fbb3bfff700 1 -- 172.21.15.88:0/1761947091 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb440037d0 con 0x7fbb54103860 2023-07-14T17:36:26.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.370+0000 7fbb3bfff700 1 -- 172.21.15.88:0/1761947091 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fbb4401b440 con 0x7fbb54103860 2023-07-14T17:36:26.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.370+0000 7fbb3bfff700 1 -- 172.21.15.88:0/1761947091 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb4401a560 con 0x7fbb54103860 2023-07-14T17:36:26.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.370+0000 7fbb595f1700 1 -- 172.21.15.88:0/1761947091 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb54199690 con 0x7fbb54103860 2023-07-14T17:36:26.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.370+0000 7fbb595f1700 1 -- 172.21.15.88:0/1761947091 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb54199ab0 con 0x7fbb54103860 2023-07-14T17:36:26.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.371+0000 7fbb3bfff700 1 -- 172.21.15.88:0/1761947091 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fbb4401a7d0 con 0x7fbb54103860 2023-07-14T17:36:26.971 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.372+0000 7fbb595f1700 1 -- 172.21.15.88:0/1761947091 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb54104e70 con 0x7fbb54103860 2023-07-14T17:36:26.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.372+0000 7fbb3bfff700 1 --2- 172.21.15.88:0/1761947091 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fbb3c038760 0x7fbb3c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:26.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.372+0000 7fbb3bfff700 1 -- 172.21.15.88:0/1761947091 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbb4404c600 con 0x7fbb54103860 2023-07-14T17:36:26.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.372+0000 7fbb527fc700 1 --2- 172.21.15.88:0/1761947091 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fbb3c038760 0x7fbb3c03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:26.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.372+0000 7fbb527fc700 1 --2- 172.21.15.88:0/1761947091 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fbb3c038760 0x7fbb3c03ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbb48006fd0 tx=0x7fbb48006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:26.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.376+0000 7fbb3bfff700 1 -- 172.21.15.88:0/1761947091 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbb440193f0 con 0x7fbb54103860 2023-07-14T17:36:26.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.562+0000 7fbb595f1700 1 -- 172.21.15.88:0/1761947091 --> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb54060a80 con 0x7fbb3c038760 2023-07-14T17:36:26.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.562+0000 7fbb3bfff700 1 -- 172.21.15.88:0/1761947091 <== mgr.14120 v2:172.21.15.88:6800/2424184187 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fbb54060a80 con 0x7fbb3c038760 2023-07-14T17:36:26.972 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.564+0000 7fbb595f1700 1 -- 172.21.15.88:0/1761947091 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fbb3c038760 msgr2=0x7fbb3c03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:26.973 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.564+0000 7fbb595f1700 1 --2- 172.21.15.88:0/1761947091 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fbb3c038760 0x7fbb3c03ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbb48006fd0 tx=0x7fbb48006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:26.973 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.564+0000 7fbb595f1700 1 -- 172.21.15.88:0/1761947091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb54103860 msgr2=0x7fbb54198f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:26.973 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.564+0000 7fbb595f1700 1 --2- 172.21.15.88:0/1761947091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb54103860 0x7fbb54198f50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbb44011dc0 tx=0x7fbb44011ea0 comp rx=0 tx=0).stop 2023-07-14T17:36:26.973 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.564+0000 7fbb595f1700 1 -- 172.21.15.88:0/1761947091 shutdown_connections 2023-07-14T17:36:26.973 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.564+0000 7fbb595f1700 1 --2- 172.21.15.88:0/1761947091 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fbb3c038760 0x7fbb3c03ac10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:26.973 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.564+0000 7fbb595f1700 1 --2- 172.21.15.88:0/1761947091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbb54103860 0x7fbb54198f50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:26.973 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.564+0000 7fbb595f1700 1 -- 172.21.15.88:0/1761947091 >> 172.21.15.88:0/1761947091 conn(0x7fbb540feef0 msgr2=0x7fbb541085c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:26.974 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.565+0000 7fbb595f1700 1 -- 172.21.15.88:0/1761947091 shutdown_connections 2023-07-14T17:36:26.974 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:26.565+0000 7fbb595f1700 1 -- 172.21.15.88:0/1761947091 wait complete. 2023-07-14T17:36:26.974 INFO:teuthology.orchestra.run.smithi088.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-14T17:36:26.974 INFO:teuthology.orchestra.run.smithi088.stdout:Adding key to root@localhost authorized_keys... 2023-07-14T17:36:26.974 INFO:teuthology.orchestra.run.smithi088.stdout:Adding host smithi088... 2023-07-14T17:36:27.894 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:27 smithi088 ceph-mon[127465]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:36:28.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:28 smithi088 ceph-mon[127465]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi088", "addr": "172.21.15.88", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:36:28.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:28 smithi088 ceph-mon[127465]: Deploying cephadm binary to smithi088 2023-07-14T17:36:30.317 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout Added host 'smithi088' with addr '172.21.15.88' 2023-07-14T17:36:30.317 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.483+0000 7f59301f7700 1 Processor -- start 2023-07-14T17:36:30.318 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.483+0000 7f59301f7700 1 -- start start 2023-07-14T17:36:30.318 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.483+0000 7f59301f7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59281045d0 0x7f59281049e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:30.318 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.483+0000 7f59301f7700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5928104f20 con 0x7f59281045d0 2023-07-14T17:36:30.318 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.483+0000 7f592df93700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59281045d0 0x7f59281049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:30.318 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.483+0000 7f592df93700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59281045d0 0x7f59281049e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40270/0 (socket says 172.21.15.88:40270) 2023-07-14T17:36:30.318 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.483+0000 7f592df93700 1 -- 172.21.15.88:0/3498543054 learned_addr learned my addr 172.21.15.88:0/3498543054 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:30.318 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.484+0000 7f592df93700 1 -- 172.21.15.88:0/3498543054 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5928105060 con 0x7f59281045d0 2023-07-14T17:36:30.319 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.484+0000 7f592df93700 1 --2- 172.21.15.88:0/3498543054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59281045d0 0x7f59281049e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f59240098a0 tx=0x7f5924009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6b0c280a2a3b969 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:30.319 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.484+0000 7f592cf91700 1 -- 172.21.15.88:0/3498543054 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5924004160 con 0x7f59281045d0 2023-07-14T17:36:30.319 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.484+0000 7f592cf91700 1 -- 172.21.15.88:0/3498543054 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f59240042c0 con 0x7f59281045d0 2023-07-14T17:36:30.319 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.484+0000 7f592cf91700 1 -- 172.21.15.88:0/3498543054 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5924011940 con 0x7f59281045d0 2023-07-14T17:36:30.319 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.484+0000 7f59301f7700 1 -- 172.21.15.88:0/3498543054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59281045d0 msgr2=0x7f59281049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:30.319 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.484+0000 7f59301f7700 1 --2- 172.21.15.88:0/3498543054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59281045d0 0x7f59281049e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f59240098a0 tx=0x7f5924009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:30.319 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.484+0000 7f59301f7700 1 -- 172.21.15.88:0/3498543054 shutdown_connections 2023-07-14T17:36:30.319 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.484+0000 7f59301f7700 1 --2- 172.21.15.88:0/3498543054 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59281045d0 0x7f59281049e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:30.320 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.484+0000 7f59301f7700 1 -- 172.21.15.88:0/3498543054 >> 172.21.15.88:0/3498543054 conn(0x7f59280feec0 msgr2=0x7f59281012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:30.320 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.484+0000 7f59301f7700 1 -- 172.21.15.88:0/3498543054 shutdown_connections 2023-07-14T17:36:30.320 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.485+0000 7f59301f7700 1 -- 172.21.15.88:0/3498543054 wait complete. 2023-07-14T17:36:30.320 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.485+0000 7f59301f7700 1 Processor -- start 2023-07-14T17:36:30.320 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.485+0000 7f59301f7700 1 -- start start 2023-07-14T17:36:30.320 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.485+0000 7f59301f7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59281045d0 0x7f5928197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:30.320 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.486+0000 7f59301f7700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5928198400 con 0x7f59281045d0 2023-07-14T17:36:30.321 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.486+0000 7f592df93700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59281045d0 0x7f5928197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:30.321 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.486+0000 7f592df93700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59281045d0 0x7f5928197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40272/0 (socket says 172.21.15.88:40272) 2023-07-14T17:36:30.321 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.486+0000 7f592df93700 1 -- 172.21.15.88:0/2618227079 learned_addr learned my addr 172.21.15.88:0/2618227079 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:30.321 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.486+0000 7f592df93700 1 -- 172.21.15.88:0/2618227079 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5924009550 con 0x7f59281045d0 2023-07-14T17:36:30.321 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.486+0000 7f592df93700 1 --2- 172.21.15.88:0/2618227079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59281045d0 0x7f5928197ec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5924000c00 tx=0x7f5924011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:30.321 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.486+0000 7f591effd700 1 -- 172.21.15.88:0/2618227079 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f59240036a0 con 0x7f59281045d0 2023-07-14T17:36:30.321 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.486+0000 7f591effd700 1 -- 172.21.15.88:0/2618227079 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f592401b440 con 0x7f59281045d0 2023-07-14T17:36:30.321 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.486+0000 7f59301f7700 1 -- 172.21.15.88:0/2618227079 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5928198600 con 0x7f59281045d0 2023-07-14T17:36:30.322 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.486+0000 7f591effd700 1 -- 172.21.15.88:0/2618227079 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f59240193f0 con 0x7f59281045d0 2023-07-14T17:36:30.322 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.486+0000 7f59301f7700 1 -- 172.21.15.88:0/2618227079 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5928199150 con 0x7f59281045d0 2023-07-14T17:36:30.322 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.487+0000 7f591effd700 1 -- 172.21.15.88:0/2618227079 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f5924003800 con 0x7f59281045d0 2023-07-14T17:36:30.322 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.487+0000 7f59301f7700 1 -- 172.21.15.88:0/2618227079 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5928061c70 con 0x7f59281045d0 2023-07-14T17:36:30.322 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.487+0000 7f591effd700 1 --2- 172.21.15.88:0/2618227079 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f5914038760 0x7f591403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:30.322 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.487+0000 7f591effd700 1 -- 172.21.15.88:0/2618227079 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f592404c570 con 0x7f59281045d0 2023-07-14T17:36:30.322 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.487+0000 7f592d792700 1 --2- 172.21.15.88:0/2618227079 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f5914038760 0x7f591403ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:30.323 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.487+0000 7f592d792700 1 --2- 172.21.15.88:0/2618227079 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f5914038760 0x7f591403ac10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5918006fd0 tx=0x7f5918006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:30.323 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.492+0000 7f591effd700 1 -- 172.21.15.88:0/2618227079 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f592401b5d0 con 0x7f59281045d0 2023-07-14T17:36:30.323 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:27.682+0000 7f59301f7700 1 -- 172.21.15.88:0/2618227079 --> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi088", "addr": "172.21.15.88", "target": ["mon-mgr", ""]}) v1 -- 0x7f59281003b0 con 0x7f5914038760 2023-07-14T17:36:30.323 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:29.954+0000 7f591effd700 1 -- 172.21.15.88:0/2618227079 <== mgr.14120 v2:172.21.15.88:6800/2424184187 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f59281003b0 con 0x7f5914038760 2023-07-14T17:36:30.323 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:29.956+0000 7f59301f7700 1 -- 172.21.15.88:0/2618227079 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f5914038760 msgr2=0x7f591403ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:30.323 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:29.956+0000 7f59301f7700 1 --2- 172.21.15.88:0/2618227079 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f5914038760 0x7f591403ac10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5918006fd0 tx=0x7f5918006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:30.323 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:29.956+0000 7f59301f7700 1 -- 172.21.15.88:0/2618227079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59281045d0 msgr2=0x7f5928197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:30.323 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:29.956+0000 7f59301f7700 1 --2- 172.21.15.88:0/2618227079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59281045d0 0x7f5928197ec0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5924000c00 tx=0x7f5924011e70 comp rx=0 tx=0).stop 2023-07-14T17:36:30.324 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:29.956+0000 7f59301f7700 1 -- 172.21.15.88:0/2618227079 shutdown_connections 2023-07-14T17:36:30.324 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:29.956+0000 7f59301f7700 1 --2- 172.21.15.88:0/2618227079 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f5914038760 0x7f591403ac10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:30.324 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:29.956+0000 7f59301f7700 1 --2- 172.21.15.88:0/2618227079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59281045d0 0x7f5928197ec0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:30.324 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:29.956+0000 7f59301f7700 1 -- 172.21.15.88:0/2618227079 >> 172.21.15.88:0/2618227079 conn(0x7f59280feec0 msgr2=0x7f59280ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:30.324 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:29.957+0000 7f59301f7700 1 -- 172.21.15.88:0/2618227079 shutdown_connections 2023-07-14T17:36:30.324 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:29.957+0000 7f59301f7700 1 -- 172.21.15.88:0/2618227079 wait complete. 2023-07-14T17:36:30.324 INFO:teuthology.orchestra.run.smithi088.stdout:Deploying unmanaged mon service... 2023-07-14T17:36:31.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:30 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' 2023-07-14T17:36:31.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:30 smithi088 ceph-mon[127465]: Added host smithi088 2023-07-14T17:36:31.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:30 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:36:31.409 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-07-14T17:36:31.410 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.822+0000 7f4b7a5dd700 1 Processor -- start 2023-07-14T17:36:31.410 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.822+0000 7f4b7a5dd700 1 -- start start 2023-07-14T17:36:31.410 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.822+0000 7f4b7a5dd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b74105cf0 0x7f4b741080d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:31.410 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.822+0000 7f4b7a5dd700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b74079d50 con 0x7f4b74105cf0 2023-07-14T17:36:31.410 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.822+0000 7f4b73fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b74105cf0 0x7f4b741080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:31.411 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.822+0000 7f4b73fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b74105cf0 0x7f4b741080d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40306/0 (socket says 172.21.15.88:40306) 2023-07-14T17:36:31.411 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.822+0000 7f4b73fff700 1 -- 172.21.15.88:0/2801355229 learned_addr learned my addr 172.21.15.88:0/2801355229 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:31.411 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.823+0000 7f4b73fff700 1 -- 172.21.15.88:0/2801355229 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b74079e90 con 0x7f4b74105cf0 2023-07-14T17:36:31.411 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.823+0000 7f4b73fff700 1 --2- 172.21.15.88:0/2801355229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b74105cf0 0x7f4b741080d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4b640098a0 tx=0x7f4b64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d13c11883859106c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:31.411 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.823+0000 7f4b72ffd700 1 -- 172.21.15.88:0/2801355229 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b64004160 con 0x7f4b74105cf0 2023-07-14T17:36:31.411 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.823+0000 7f4b72ffd700 1 -- 172.21.15.88:0/2801355229 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f4b640042c0 con 0x7f4b74105cf0 2023-07-14T17:36:31.411 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.823+0000 7f4b72ffd700 1 -- 172.21.15.88:0/2801355229 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b64011940 con 0x7f4b74105cf0 2023-07-14T17:36:31.412 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.823+0000 7f4b7a5dd700 1 -- 172.21.15.88:0/2801355229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b74105cf0 msgr2=0x7f4b741080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:31.412 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.823+0000 7f4b7a5dd700 1 --2- 172.21.15.88:0/2801355229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b74105cf0 0x7f4b741080d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4b640098a0 tx=0x7f4b64009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:31.412 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.823+0000 7f4b7a5dd700 1 -- 172.21.15.88:0/2801355229 shutdown_connections 2023-07-14T17:36:31.412 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.823+0000 7f4b7a5dd700 1 --2- 172.21.15.88:0/2801355229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b74105cf0 0x7f4b741080d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:31.412 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.823+0000 7f4b7a5dd700 1 -- 172.21.15.88:0/2801355229 >> 172.21.15.88:0/2801355229 conn(0x7f4b740ff750 msgr2=0x7f4b74101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:31.413 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.824+0000 7f4b7a5dd700 1 -- 172.21.15.88:0/2801355229 shutdown_connections 2023-07-14T17:36:31.413 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.824+0000 7f4b7a5dd700 1 -- 172.21.15.88:0/2801355229 wait complete. 2023-07-14T17:36:31.413 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.824+0000 7f4b7a5dd700 1 Processor -- start 2023-07-14T17:36:31.414 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.824+0000 7f4b7a5dd700 1 -- start start 2023-07-14T17:36:31.414 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.824+0000 7f4b7a5dd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b74105cf0 0x7f4b74198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:31.414 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.824+0000 7f4b7a5dd700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b74199460 con 0x7f4b74105cf0 2023-07-14T17:36:31.414 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.824+0000 7f4b73fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b74105cf0 0x7f4b74198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:31.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.825+0000 7f4b73fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b74105cf0 0x7f4b74198f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40308/0 (socket says 172.21.15.88:40308) 2023-07-14T17:36:31.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.825+0000 7f4b73fff700 1 -- 172.21.15.88:0/641947724 learned_addr learned my addr 172.21.15.88:0/641947724 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:31.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.825+0000 7f4b73fff700 1 -- 172.21.15.88:0/641947724 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b64009550 con 0x7f4b74105cf0 2023-07-14T17:36:31.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.825+0000 7f4b73fff700 1 --2- 172.21.15.88:0/641947724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b74105cf0 0x7f4b74198f20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4b64006af0 tx=0x7f4b64011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:31.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.825+0000 7f4b717fa700 1 -- 172.21.15.88:0/641947724 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b64003bf0 con 0x7f4b74105cf0 2023-07-14T17:36:31.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.825+0000 7f4b717fa700 1 -- 172.21.15.88:0/641947724 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f4b6401b8b0 con 0x7f4b74105cf0 2023-07-14T17:36:31.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.825+0000 7f4b7a5dd700 1 -- 172.21.15.88:0/641947724 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b74199660 con 0x7f4b74105cf0 2023-07-14T17:36:31.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.825+0000 7f4b717fa700 1 -- 172.21.15.88:0/641947724 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b640198b0 con 0x7f4b74105cf0 2023-07-14T17:36:31.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.825+0000 7f4b7a5dd700 1 -- 172.21.15.88:0/641947724 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b74199a80 con 0x7f4b74105cf0 2023-07-14T17:36:31.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.826+0000 7f4b717fa700 1 -- 172.21.15.88:0/641947724 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f4b64019b70 con 0x7f4b74105cf0 2023-07-14T17:36:31.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.826+0000 7f4b7a5dd700 1 -- 172.21.15.88:0/641947724 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b74109280 con 0x7f4b74105cf0 2023-07-14T17:36:31.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.826+0000 7f4b717fa700 1 --2- 172.21.15.88:0/641947724 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4b5c0386c0 0x7f4b5c03ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:31.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.826+0000 7f4b717fa700 1 -- 172.21.15.88:0/641947724 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4b6404cbc0 con 0x7f4b74105cf0 2023-07-14T17:36:31.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.826+0000 7f4b737fe700 1 --2- 172.21.15.88:0/641947724 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4b5c0386c0 0x7f4b5c03ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:31.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.827+0000 7f4b737fe700 1 --2- 172.21.15.88:0/641947724 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4b5c0386c0 0x7f4b5c03ab70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4b68006fd0 tx=0x7f4b68006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:31.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:30.831+0000 7f4b717fa700 1 -- 172.21.15.88:0/641947724 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4b640189a0 con 0x7f4b74105cf0 2023-07-14T17:36:31.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.031+0000 7f4b7a5dd700 1 -- 172.21.15.88:0/641947724 --> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f4b7419ab50 con 0x7f4b5c0386c0 2023-07-14T17:36:31.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.036+0000 7f4b717fa700 1 -- 172.21.15.88:0/641947724 <== mgr.14120 v2:172.21.15.88:6800/2424184187 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4b7419ab50 con 0x7f4b5c0386c0 2023-07-14T17:36:31.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.038+0000 7f4b7a5dd700 1 -- 172.21.15.88:0/641947724 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4b5c0386c0 msgr2=0x7f4b5c03ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:31.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.038+0000 7f4b7a5dd700 1 --2- 172.21.15.88:0/641947724 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4b5c0386c0 0x7f4b5c03ab70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4b68006fd0 tx=0x7f4b68006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:31.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.038+0000 7f4b7a5dd700 1 -- 172.21.15.88:0/641947724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b74105cf0 msgr2=0x7f4b74198f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:31.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.038+0000 7f4b7a5dd700 1 --2- 172.21.15.88:0/641947724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b74105cf0 0x7f4b74198f20 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4b64006af0 tx=0x7f4b64011a30 comp rx=0 tx=0).stop 2023-07-14T17:36:31.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.038+0000 7f4b7a5dd700 1 -- 172.21.15.88:0/641947724 shutdown_connections 2023-07-14T17:36:31.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.038+0000 7f4b7a5dd700 1 --2- 172.21.15.88:0/641947724 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4b5c0386c0 0x7f4b5c03ab70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:31.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.038+0000 7f4b7a5dd700 1 --2- 172.21.15.88:0/641947724 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b74105cf0 0x7f4b74198f20 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:31.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.038+0000 7f4b7a5dd700 1 -- 172.21.15.88:0/641947724 >> 172.21.15.88:0/641947724 conn(0x7f4b740ff750 msgr2=0x7f4b741004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:31.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.038+0000 7f4b7a5dd700 1 -- 172.21.15.88:0/641947724 shutdown_connections 2023-07-14T17:36:31.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.038+0000 7f4b7a5dd700 1 -- 172.21.15.88:0/641947724 wait complete. 2023-07-14T17:36:31.419 INFO:teuthology.orchestra.run.smithi088.stdout:Deploying unmanaged mgr service... 2023-07-14T17:36:32.313 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:32 smithi088 ceph-mon[127465]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:36:32.314 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:32 smithi088 ceph-mon[127465]: Saving service mon spec with placement count:5 2023-07-14T17:36:32.314 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:32 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' 2023-07-14T17:36:32.502 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-07-14T17:36:32.502 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.958+0000 7fe7cf0b8700 1 Processor -- start 2023-07-14T17:36:32.502 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.958+0000 7fe7cf0b8700 1 -- start start 2023-07-14T17:36:32.502 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.959+0000 7fe7cf0b8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7c81040e0 0x7fe7c81044f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:32.502 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.959+0000 7fe7cf0b8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c8104a30 con 0x7fe7c81040e0 2023-07-14T17:36:32.503 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.959+0000 7fe7cce54700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7c81040e0 0x7fe7c81044f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:32.503 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.959+0000 7fe7cce54700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7c81040e0 0x7fe7c81044f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40312/0 (socket says 172.21.15.88:40312) 2023-07-14T17:36:32.503 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.959+0000 7fe7cce54700 1 -- 172.21.15.88:0/1638395711 learned_addr learned my addr 172.21.15.88:0/1638395711 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:32.503 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.959+0000 7fe7cce54700 1 -- 172.21.15.88:0/1638395711 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c8104b70 con 0x7fe7c81040e0 2023-07-14T17:36:32.503 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.960+0000 7fe7cce54700 1 --2- 172.21.15.88:0/1638395711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7c81040e0 0x7fe7c81044f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0009e80 tx=0x7fe7c000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27f5311304fd94c7 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:32.503 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.960+0000 7fe7c77fe700 1 -- 172.21.15.88:0/1638395711 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe7c000e070 con 0x7fe7c81040e0 2023-07-14T17:36:32.503 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.960+0000 7fe7c77fe700 1 -- 172.21.15.88:0/1638395711 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fe7c0003e90 con 0x7fe7c81040e0 2023-07-14T17:36:32.504 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.960+0000 7fe7c77fe700 1 -- 172.21.15.88:0/1638395711 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe7c0004360 con 0x7fe7c81040e0 2023-07-14T17:36:32.504 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.960+0000 7fe7cf0b8700 1 -- 172.21.15.88:0/1638395711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7c81040e0 msgr2=0x7fe7c81044f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:32.504 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.960+0000 7fe7cf0b8700 1 --2- 172.21.15.88:0/1638395711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7c81040e0 0x7fe7c81044f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe7c0009e80 tx=0x7fe7c000b330 comp rx=0 tx=0).stop 2023-07-14T17:36:32.504 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.960+0000 7fe7cf0b8700 1 -- 172.21.15.88:0/1638395711 shutdown_connections 2023-07-14T17:36:32.504 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.960+0000 7fe7cf0b8700 1 --2- 172.21.15.88:0/1638395711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7c81040e0 0x7fe7c81044f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:32.504 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.960+0000 7fe7cf0b8700 1 -- 172.21.15.88:0/1638395711 >> 172.21.15.88:0/1638395711 conn(0x7fe7c80ff750 msgr2=0x7fe7c8101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:32.504 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.960+0000 7fe7cf0b8700 1 -- 172.21.15.88:0/1638395711 shutdown_connections 2023-07-14T17:36:32.505 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.960+0000 7fe7cf0b8700 1 -- 172.21.15.88:0/1638395711 wait complete. 2023-07-14T17:36:32.505 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.961+0000 7fe7cf0b8700 1 Processor -- start 2023-07-14T17:36:32.505 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.961+0000 7fe7cf0b8700 1 -- start start 2023-07-14T17:36:32.505 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.961+0000 7fe7cf0b8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7c81040e0 0x7fe7c8197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:32.505 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.961+0000 7fe7cf0b8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c8198410 con 0x7fe7c81040e0 2023-07-14T17:36:32.505 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.961+0000 7fe7cce54700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7c81040e0 0x7fe7c8197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:32.505 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.961+0000 7fe7cce54700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7c81040e0 0x7fe7c8197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40314/0 (socket says 172.21.15.88:40314) 2023-07-14T17:36:32.505 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.961+0000 7fe7cce54700 1 -- 172.21.15.88:0/3397114916 learned_addr learned my addr 172.21.15.88:0/3397114916 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:32.506 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.961+0000 7fe7cce54700 1 -- 172.21.15.88:0/3397114916 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c0009b30 con 0x7fe7c81040e0 2023-07-14T17:36:32.506 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.962+0000 7fe7cce54700 1 --2- 172.21.15.88:0/3397114916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7c81040e0 0x7fe7c8197ed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe7c00072a0 tx=0x7fe7c0003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:32.506 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.962+0000 7fe7c5ffb700 1 -- 172.21.15.88:0/3397114916 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe7c000e040 con 0x7fe7c81040e0 2023-07-14T17:36:32.506 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.962+0000 7fe7c5ffb700 1 -- 172.21.15.88:0/3397114916 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fe7c00128a0 con 0x7fe7c81040e0 2023-07-14T17:36:32.506 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.962+0000 7fe7c5ffb700 1 -- 172.21.15.88:0/3397114916 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe7c001c6b0 con 0x7fe7c81040e0 2023-07-14T17:36:32.506 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.962+0000 7fe7cf0b8700 1 -- 172.21.15.88:0/3397114916 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7c8198610 con 0x7fe7c81040e0 2023-07-14T17:36:32.506 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.962+0000 7fe7cf0b8700 1 -- 172.21.15.88:0/3397114916 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7c8199160 con 0x7fe7c81040e0 2023-07-14T17:36:32.507 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.963+0000 7fe7c5ffb700 1 -- 172.21.15.88:0/3397114916 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fe7c0012430 con 0x7fe7c81040e0 2023-07-14T17:36:32.507 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.963+0000 7fe7c5ffb700 1 --2- 172.21.15.88:0/3397114916 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fe7b4038740 0x7fe7b403abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:32.507 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.963+0000 7fe7cf0b8700 1 -- 172.21.15.88:0/3397114916 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7c8061c70 con 0x7fe7c81040e0 2023-07-14T17:36:32.507 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.963+0000 7fe7c5ffb700 1 -- 172.21.15.88:0/3397114916 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe7c004c630 con 0x7fe7c81040e0 2023-07-14T17:36:32.507 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.963+0000 7fe7c7fff700 1 --2- 172.21.15.88:0/3397114916 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fe7b4038740 0x7fe7b403abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:32.507 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.963+0000 7fe7c7fff700 1 --2- 172.21.15.88:0/3397114916 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fe7b4038740 0x7fe7b403abf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0006fd0 tx=0x7fe7b0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:32.507 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:31.968+0000 7fe7c5ffb700 1 -- 172.21.15.88:0/3397114916 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe7c0012a10 con 0x7fe7c81040e0 2023-07-14T17:36:32.507 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:32.171+0000 7fe7cf0b8700 1 -- 172.21.15.88:0/3397114916 --> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fe7c8100ba0 con 0x7fe7b4038740 2023-07-14T17:36:32.508 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:32.175+0000 7fe7c5ffb700 1 -- 172.21.15.88:0/3397114916 <== mgr.14120 v2:172.21.15.88:6800/2424184187 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fe7c8100ba0 con 0x7fe7b4038740 2023-07-14T17:36:32.508 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:32.177+0000 7fe7cf0b8700 1 -- 172.21.15.88:0/3397114916 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fe7b4038740 msgr2=0x7fe7b403abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:32.508 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:32.177+0000 7fe7cf0b8700 1 --2- 172.21.15.88:0/3397114916 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fe7b4038740 0x7fe7b403abf0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0006fd0 tx=0x7fe7b0006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:32.508 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:32.177+0000 7fe7cf0b8700 1 -- 172.21.15.88:0/3397114916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7c81040e0 msgr2=0x7fe7c8197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:32.508 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:32.177+0000 7fe7cf0b8700 1 --2- 172.21.15.88:0/3397114916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7c81040e0 0x7fe7c8197ed0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe7c00072a0 tx=0x7fe7c0003d00 comp rx=0 tx=0).stop 2023-07-14T17:36:32.508 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:32.177+0000 7fe7cf0b8700 1 -- 172.21.15.88:0/3397114916 shutdown_connections 2023-07-14T17:36:32.508 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:32.177+0000 7fe7cf0b8700 1 --2- 172.21.15.88:0/3397114916 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fe7b4038740 0x7fe7b403abf0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:32.509 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:32.177+0000 7fe7cf0b8700 1 --2- 172.21.15.88:0/3397114916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7c81040e0 0x7fe7c8197ed0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:32.509 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:32.177+0000 7fe7cf0b8700 1 -- 172.21.15.88:0/3397114916 >> 172.21.15.88:0/3397114916 conn(0x7fe7c80ff750 msgr2=0x7fe7c8100490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:32.509 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:32.177+0000 7fe7cf0b8700 1 -- 172.21.15.88:0/3397114916 shutdown_connections 2023-07-14T17:36:32.509 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:32.178+0000 7fe7cf0b8700 1 -- 172.21.15.88:0/3397114916 wait complete. 2023-07-14T17:36:33.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:33 smithi088 ceph-mon[127465]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:36:33.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:33 smithi088 ceph-mon[127465]: Saving service mgr spec with placement count:2 2023-07-14T17:36:33.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:33 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' 2023-07-14T17:36:33.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:33 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' 2023-07-14T17:36:34.548 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.958+0000 7fc57b50e700 1 Processor -- start 2023-07-14T17:36:34.548 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.958+0000 7fc57b50e700 1 -- start start 2023-07-14T17:36:34.548 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.958+0000 7fc57b50e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc574105d40 0x7fc574108120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:34.549 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.958+0000 7fc57b50e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc574079d10 con 0x7fc574105d40 2023-07-14T17:36:34.549 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.958+0000 7fc5792aa700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc574105d40 0x7fc574108120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:34.549 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.958+0000 7fc5792aa700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc574105d40 0x7fc574108120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40318/0 (socket says 172.21.15.88:40318) 2023-07-14T17:36:34.549 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.958+0000 7fc5792aa700 1 -- 172.21.15.88:0/1990382727 learned_addr learned my addr 172.21.15.88:0/1990382727 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:34.549 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.959+0000 7fc5792aa700 1 -- 172.21.15.88:0/1990382727 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc574079e50 con 0x7fc574105d40 2023-07-14T17:36:34.549 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.959+0000 7fc5792aa700 1 --2- 172.21.15.88:0/1990382727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc574105d40 0x7fc574108120 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc5700098a0 tx=0x7fc570009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d60e1cf118e71cb8 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:34.549 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.959+0000 7fc56bfff700 1 -- 172.21.15.88:0/1990382727 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc570004160 con 0x7fc574105d40 2023-07-14T17:36:34.549 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.959+0000 7fc56bfff700 1 -- 172.21.15.88:0/1990382727 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc5700042c0 con 0x7fc574105d40 2023-07-14T17:36:34.550 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.959+0000 7fc56bfff700 1 -- 172.21.15.88:0/1990382727 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5700118e0 con 0x7fc574105d40 2023-07-14T17:36:34.550 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.960+0000 7fc57b50e700 1 -- 172.21.15.88:0/1990382727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc574105d40 msgr2=0x7fc574108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:34.550 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.960+0000 7fc57b50e700 1 --2- 172.21.15.88:0/1990382727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc574105d40 0x7fc574108120 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc5700098a0 tx=0x7fc570009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:34.550 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.960+0000 7fc57b50e700 1 -- 172.21.15.88:0/1990382727 shutdown_connections 2023-07-14T17:36:34.550 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.960+0000 7fc57b50e700 1 --2- 172.21.15.88:0/1990382727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc574105d40 0x7fc574108120 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:34.550 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.960+0000 7fc57b50e700 1 -- 172.21.15.88:0/1990382727 >> 172.21.15.88:0/1990382727 conn(0x7fc5740ff7a0 msgr2=0x7fc574101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:34.550 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.960+0000 7fc57b50e700 1 -- 172.21.15.88:0/1990382727 shutdown_connections 2023-07-14T17:36:34.551 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.960+0000 7fc57b50e700 1 -- 172.21.15.88:0/1990382727 wait complete. 2023-07-14T17:36:34.551 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.961+0000 7fc57b50e700 1 Processor -- start 2023-07-14T17:36:34.551 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.961+0000 7fc57b50e700 1 -- start start 2023-07-14T17:36:34.551 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.961+0000 7fc57b50e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc574105d40 0x7fc574198ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:34.551 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.961+0000 7fc57b50e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc574199530 con 0x7fc574105d40 2023-07-14T17:36:34.551 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.961+0000 7fc5792aa700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc574105d40 0x7fc574198ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:34.551 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.961+0000 7fc5792aa700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc574105d40 0x7fc574198ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40320/0 (socket says 172.21.15.88:40320) 2023-07-14T17:36:34.551 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.961+0000 7fc5792aa700 1 -- 172.21.15.88:0/365524577 learned_addr learned my addr 172.21.15.88:0/365524577 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:34.552 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.961+0000 7fc5792aa700 1 -- 172.21.15.88:0/365524577 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc570009550 con 0x7fc574105d40 2023-07-14T17:36:34.552 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.961+0000 7fc5792aa700 1 --2- 172.21.15.88:0/365524577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc574105d40 0x7fc574198ff0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc570006af0 tx=0x7fc570011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:34.552 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.961+0000 7fc56a7fc700 1 -- 172.21.15.88:0/365524577 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5700037d0 con 0x7fc574105d40 2023-07-14T17:36:34.552 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.961+0000 7fc56a7fc700 1 -- 172.21.15.88:0/365524577 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc57001b440 con 0x7fc574105d40 2023-07-14T17:36:34.552 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.961+0000 7fc57b50e700 1 -- 172.21.15.88:0/365524577 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc574199790 con 0x7fc574105d40 2023-07-14T17:36:34.552 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.962+0000 7fc56a7fc700 1 -- 172.21.15.88:0/365524577 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc570019560 con 0x7fc574105d40 2023-07-14T17:36:34.552 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.962+0000 7fc57b50e700 1 -- 172.21.15.88:0/365524577 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc57419a3c0 con 0x7fc574105d40 2023-07-14T17:36:34.553 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.962+0000 7fc56a7fc700 1 -- 172.21.15.88:0/365524577 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fc5700197d0 con 0x7fc574105d40 2023-07-14T17:36:34.553 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.962+0000 7fc57b50e700 1 -- 172.21.15.88:0/365524577 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc574061c70 con 0x7fc574105d40 2023-07-14T17:36:34.553 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.963+0000 7fc56a7fc700 1 --2- 172.21.15.88:0/365524577 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fc560038710 0x7fc56003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:34.553 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.963+0000 7fc56a7fc700 1 -- 172.21.15.88:0/365524577 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc57004c4e0 con 0x7fc574105d40 2023-07-14T17:36:34.553 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.963+0000 7fc578aa9700 1 --2- 172.21.15.88:0/365524577 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fc560038710 0x7fc56003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:34.553 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.963+0000 7fc578aa9700 1 --2- 172.21.15.88:0/365524577 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fc560038710 0x7fc56003abc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc564006fd0 tx=0x7fc564006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:34.553 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:33.970+0000 7fc56a7fc700 1 -- 172.21.15.88:0/365524577 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc570018db0 con 0x7fc574105d40 2023-07-14T17:36:34.553 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:34.164+0000 7fc57b50e700 1 -- 172.21.15.88:0/365524577 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fc57419a960 con 0x7fc574105d40 2023-07-14T17:36:34.554 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:34.167+0000 7fc56a7fc700 1 -- 172.21.15.88:0/365524577 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fc570018350 con 0x7fc574105d40 2023-07-14T17:36:34.554 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:34.170+0000 7fc57b50e700 1 -- 172.21.15.88:0/365524577 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fc560038710 msgr2=0x7fc56003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:34.554 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:34.170+0000 7fc57b50e700 1 --2- 172.21.15.88:0/365524577 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fc560038710 0x7fc56003abc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc564006fd0 tx=0x7fc564006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:34.554 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:34.170+0000 7fc57b50e700 1 -- 172.21.15.88:0/365524577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc574105d40 msgr2=0x7fc574198ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:34.554 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:34.170+0000 7fc57b50e700 1 --2- 172.21.15.88:0/365524577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc574105d40 0x7fc574198ff0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc570006af0 tx=0x7fc570011e80 comp rx=0 tx=0).stop 2023-07-14T17:36:34.554 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:34.170+0000 7fc57b50e700 1 -- 172.21.15.88:0/365524577 shutdown_connections 2023-07-14T17:36:34.554 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:34.170+0000 7fc57b50e700 1 --2- 172.21.15.88:0/365524577 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7fc560038710 0x7fc56003abc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:34.555 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:34.170+0000 7fc57b50e700 1 --2- 172.21.15.88:0/365524577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc574105d40 0x7fc574198ff0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:34.555 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:34.170+0000 7fc57b50e700 1 -- 172.21.15.88:0/365524577 >> 172.21.15.88:0/365524577 conn(0x7fc5740ff7a0 msgr2=0x7fc574100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:34.555 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:34.170+0000 7fc57b50e700 1 -- 172.21.15.88:0/365524577 shutdown_connections 2023-07-14T17:36:34.555 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:34.170+0000 7fc57b50e700 1 -- 172.21.15.88:0/365524577 wait complete. 2023-07-14T17:36:34.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:34 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' 2023-07-14T17:36:34.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:34 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/365524577' entity='client.admin' 2023-07-14T17:36:35.840 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.248+0000 7f9bda683700 1 Processor -- start 2023-07-14T17:36:35.840 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.248+0000 7f9bda683700 1 -- start start 2023-07-14T17:36:35.840 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.248+0000 7f9bda683700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd41040e0 0x7f9bd41044f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:35.840 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.248+0000 7f9bda683700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd4104a30 con 0x7f9bd41040e0 2023-07-14T17:36:35.840 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.248+0000 7f9bd3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd41040e0 0x7f9bd41044f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:35.841 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.248+0000 7f9bd3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd41040e0 0x7f9bd41044f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40324/0 (socket says 172.21.15.88:40324) 2023-07-14T17:36:35.841 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.248+0000 7f9bd3fff700 1 -- 172.21.15.88:0/3147629543 learned_addr learned my addr 172.21.15.88:0/3147629543 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:35.841 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.249+0000 7f9bd3fff700 1 -- 172.21.15.88:0/3147629543 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd4104b70 con 0x7f9bd41040e0 2023-07-14T17:36:35.841 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.249+0000 7f9bd3fff700 1 --2- 172.21.15.88:0/3147629543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd41040e0 0x7f9bd41044f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9bc40098a0 tx=0x7f9bc4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c6b76a594ff322 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:35.841 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.249+0000 7f9bd2ffd700 1 -- 172.21.15.88:0/3147629543 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bc4004160 con 0x7f9bd41040e0 2023-07-14T17:36:35.841 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.249+0000 7f9bd2ffd700 1 -- 172.21.15.88:0/3147629543 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9bc40042c0 con 0x7f9bd41040e0 2023-07-14T17:36:35.845 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.249+0000 7f9bd2ffd700 1 -- 172.21.15.88:0/3147629543 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bc40118e0 con 0x7f9bd41040e0 2023-07-14T17:36:35.846 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.250+0000 7f9bda683700 1 -- 172.21.15.88:0/3147629543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd41040e0 msgr2=0x7f9bd41044f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:35.846 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.250+0000 7f9bda683700 1 --2- 172.21.15.88:0/3147629543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd41040e0 0x7f9bd41044f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9bc40098a0 tx=0x7f9bc4009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:35.846 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.250+0000 7f9bda683700 1 -- 172.21.15.88:0/3147629543 shutdown_connections 2023-07-14T17:36:35.846 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.250+0000 7f9bda683700 1 --2- 172.21.15.88:0/3147629543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd41040e0 0x7f9bd41044f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:35.846 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.250+0000 7f9bda683700 1 -- 172.21.15.88:0/3147629543 >> 172.21.15.88:0/3147629543 conn(0x7f9bd40ff750 msgr2=0x7f9bd4101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:35.847 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.250+0000 7f9bda683700 1 -- 172.21.15.88:0/3147629543 shutdown_connections 2023-07-14T17:36:35.847 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.250+0000 7f9bda683700 1 -- 172.21.15.88:0/3147629543 wait complete. 2023-07-14T17:36:35.847 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.250+0000 7f9bda683700 1 Processor -- start 2023-07-14T17:36:35.847 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.250+0000 7f9bda683700 1 -- start start 2023-07-14T17:36:35.847 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.251+0000 7f9bda683700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd41040e0 0x7f9bd419e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:35.847 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.251+0000 7f9bda683700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd4199a60 con 0x7f9bd41040e0 2023-07-14T17:36:35.847 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.251+0000 7f9bd3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd41040e0 0x7f9bd419e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:35.848 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.251+0000 7f9bd3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd41040e0 0x7f9bd419e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40326/0 (socket says 172.21.15.88:40326) 2023-07-14T17:36:35.848 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.251+0000 7f9bd3fff700 1 -- 172.21.15.88:0/1502987175 learned_addr learned my addr 172.21.15.88:0/1502987175 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:35.848 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.251+0000 7f9bd3fff700 1 -- 172.21.15.88:0/1502987175 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc4009550 con 0x7f9bd41040e0 2023-07-14T17:36:35.848 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.251+0000 7f9bd3fff700 1 --2- 172.21.15.88:0/1502987175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd41040e0 0x7f9bd419e070 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9bc40118f0 tx=0x7f9bc40119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:35.848 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.252+0000 7f9bd17fa700 1 -- 172.21.15.88:0/1502987175 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bc40036a0 con 0x7f9bd41040e0 2023-07-14T17:36:35.848 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.252+0000 7f9bd17fa700 1 -- 172.21.15.88:0/1502987175 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9bc401b440 con 0x7f9bd41040e0 2023-07-14T17:36:35.848 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.252+0000 7f9bd17fa700 1 -- 172.21.15.88:0/1502987175 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bc40193f0 con 0x7f9bd41040e0 2023-07-14T17:36:35.849 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.252+0000 7f9bda683700 1 -- 172.21.15.88:0/1502987175 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd4079ce0 con 0x7f9bd41040e0 2023-07-14T17:36:35.849 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.252+0000 7f9bda683700 1 -- 172.21.15.88:0/1502987175 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd419e780 con 0x7f9bd41040e0 2023-07-14T17:36:35.849 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.253+0000 7f9bd17fa700 1 -- 172.21.15.88:0/1502987175 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f9bc40196b0 con 0x7f9bd41040e0 2023-07-14T17:36:35.849 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.253+0000 7f9bda683700 1 -- 172.21.15.88:0/1502987175 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd4061c70 con 0x7f9bd41040e0 2023-07-14T17:36:35.849 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.253+0000 7f9bd17fa700 1 --2- 172.21.15.88:0/1502987175 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f9bbc038710 0x7f9bbc03abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:35.849 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.253+0000 7f9bd17fa700 1 -- 172.21.15.88:0/1502987175 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9bc404c7c0 con 0x7f9bd41040e0 2023-07-14T17:36:35.849 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.253+0000 7f9bd37fe700 1 --2- 172.21.15.88:0/1502987175 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f9bbc038710 0x7f9bbc03abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:35.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.253+0000 7f9bd37fe700 1 --2- 172.21.15.88:0/1502987175 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f9bbc038710 0x7f9bbc03abc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8006fd0 tx=0x7f9bc8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:35.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.258+0000 7f9bd17fa700 1 -- 172.21.15.88:0/1502987175 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9bc4023770 con 0x7f9bd41040e0 2023-07-14T17:36:35.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.454+0000 7f9bda683700 1 -- 172.21.15.88:0/1502987175 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f9bd4068570 con 0x7f9bd41040e0 2023-07-14T17:36:35.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.459+0000 7f9bd17fa700 1 -- 172.21.15.88:0/1502987175 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f9bc40233b0 con 0x7f9bd41040e0 2023-07-14T17:36:35.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.462+0000 7f9bda683700 1 -- 172.21.15.88:0/1502987175 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f9bbc038710 msgr2=0x7f9bbc03abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:35.850 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.462+0000 7f9bda683700 1 --2- 172.21.15.88:0/1502987175 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f9bbc038710 0x7f9bbc03abc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8006fd0 tx=0x7f9bc8006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:35.851 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.462+0000 7f9bda683700 1 -- 172.21.15.88:0/1502987175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd41040e0 msgr2=0x7f9bd419e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:35.851 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.462+0000 7f9bda683700 1 --2- 172.21.15.88:0/1502987175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd41040e0 0x7f9bd419e070 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9bc40118f0 tx=0x7f9bc40119d0 comp rx=0 tx=0).stop 2023-07-14T17:36:35.851 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.462+0000 7f9bda683700 1 -- 172.21.15.88:0/1502987175 shutdown_connections 2023-07-14T17:36:35.851 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.462+0000 7f9bda683700 1 --2- 172.21.15.88:0/1502987175 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f9bbc038710 0x7f9bbc03abc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:35.851 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.462+0000 7f9bda683700 1 --2- 172.21.15.88:0/1502987175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9bd41040e0 0x7f9bd419e070 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:35.852 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.462+0000 7f9bda683700 1 -- 172.21.15.88:0/1502987175 >> 172.21.15.88:0/1502987175 conn(0x7f9bd40ff750 msgr2=0x7f9bd4100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:35.852 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.462+0000 7f9bda683700 1 -- 172.21.15.88:0/1502987175 shutdown_connections 2023-07-14T17:36:35.852 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:35.462+0000 7f9bda683700 1 -- 172.21.15.88:0/1502987175 wait complete. 2023-07-14T17:36:35.852 INFO:teuthology.orchestra.run.smithi088.stdout:Enabling the dashboard module... 2023-07-14T17:36:36.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:36 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' 2023-07-14T17:36:36.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:36 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1502987175' entity='client.admin' 2023-07-14T17:36:36.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:36 smithi088 ceph-mon[127465]: from='mgr.14120 172.21.15.88:0/1339784694' entity='mgr.x' 2023-07-14T17:36:37.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:37 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2431345111' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-14T17:36:37.527 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:37 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: ignoring --setuser ceph since I am not root 2023-07-14T17:36:37.527 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:37 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: ignoring --setgroup ceph since I am not root 2023-07-14T17:36:37.527 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:37 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:37.368+0000 7f452d59d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T17:36:37.527 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:37 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:37.449+0000 7f452d59d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T17:36:37.527 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:37 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:37.539+0000 7f452d59d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T17:36:37.699 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.377+0000 7f0058676700 1 Processor -- start 2023-07-14T17:36:37.700 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.377+0000 7f0058676700 1 -- start start 2023-07-14T17:36:37.700 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.377+0000 7f0058676700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00501067f0 0x7f0050106c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:37.700 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.377+0000 7f0058676700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0050107140 con 0x7f00501067f0 2023-07-14T17:36:37.700 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.377+0000 7f0056412700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00501067f0 0x7f0050106c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:37.700 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.377+0000 7f0056412700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00501067f0 0x7f0050106c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40330/0 (socket says 172.21.15.88:40330) 2023-07-14T17:36:37.700 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.377+0000 7f0056412700 1 -- 172.21.15.88:0/1182883867 learned_addr learned my addr 172.21.15.88:0/1182883867 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:37.701 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.378+0000 7f0056412700 1 -- 172.21.15.88:0/1182883867 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0050107280 con 0x7f00501067f0 2023-07-14T17:36:37.701 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.378+0000 7f0056412700 1 --2- 172.21.15.88:0/1182883867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00501067f0 0x7f0050106c00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f004c0098a0 tx=0x7f004c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e601519d292b7ea server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:37.701 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.378+0000 7f0055410700 1 -- 172.21.15.88:0/1182883867 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f004c004160 con 0x7f00501067f0 2023-07-14T17:36:37.701 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.378+0000 7f0055410700 1 -- 172.21.15.88:0/1182883867 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f004c0042c0 con 0x7f00501067f0 2023-07-14T17:36:37.701 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.378+0000 7f0055410700 1 -- 172.21.15.88:0/1182883867 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f004c0118e0 con 0x7f00501067f0 2023-07-14T17:36:37.701 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.378+0000 7f0058676700 1 -- 172.21.15.88:0/1182883867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00501067f0 msgr2=0x7f0050106c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:37.701 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.378+0000 7f0058676700 1 --2- 172.21.15.88:0/1182883867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00501067f0 0x7f0050106c00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f004c0098a0 tx=0x7f004c009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:37.702 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.379+0000 7f0058676700 1 -- 172.21.15.88:0/1182883867 shutdown_connections 2023-07-14T17:36:37.702 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.379+0000 7f0058676700 1 --2- 172.21.15.88:0/1182883867 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00501067f0 0x7f0050106c00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:37.702 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.379+0000 7f0058676700 1 -- 172.21.15.88:0/1182883867 >> 172.21.15.88:0/1182883867 conn(0x7f00500feec0 msgr2=0x7f00501012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:37.702 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.379+0000 7f0058676700 1 -- 172.21.15.88:0/1182883867 shutdown_connections 2023-07-14T17:36:37.702 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.379+0000 7f0058676700 1 -- 172.21.15.88:0/1182883867 wait complete. 2023-07-14T17:36:37.702 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.379+0000 7f0058676700 1 Processor -- start 2023-07-14T17:36:37.702 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.379+0000 7f0058676700 1 -- start start 2023-07-14T17:36:37.702 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.380+0000 7f0058676700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00501067f0 0x7f0050198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:37.703 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.380+0000 7f0058676700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00501994c0 con 0x7f00501067f0 2023-07-14T17:36:37.703 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.380+0000 7f0056412700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00501067f0 0x7f0050198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:37.703 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.380+0000 7f0056412700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00501067f0 0x7f0050198f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40332/0 (socket says 172.21.15.88:40332) 2023-07-14T17:36:37.703 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.380+0000 7f0056412700 1 -- 172.21.15.88:0/2431345111 learned_addr learned my addr 172.21.15.88:0/2431345111 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:37.703 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.380+0000 7f0056412700 1 -- 172.21.15.88:0/2431345111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f004c009550 con 0x7f00501067f0 2023-07-14T17:36:37.703 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.380+0000 7f0056412700 1 --2- 172.21.15.88:0/2431345111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00501067f0 0x7f0050198f80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f004c006af0 tx=0x7f004c011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:37.703 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.380+0000 7f00477fe700 1 -- 172.21.15.88:0/2431345111 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f004c0037d0 con 0x7f00501067f0 2023-07-14T17:36:37.703 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.380+0000 7f00477fe700 1 -- 172.21.15.88:0/2431345111 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f004c01b440 con 0x7f00501067f0 2023-07-14T17:36:37.704 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.380+0000 7f0058676700 1 -- 172.21.15.88:0/2431345111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00501996c0 con 0x7f00501067f0 2023-07-14T17:36:37.704 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.380+0000 7f00477fe700 1 -- 172.21.15.88:0/2431345111 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f004c0193d0 con 0x7f00501067f0 2023-07-14T17:36:37.704 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.380+0000 7f0058676700 1 -- 172.21.15.88:0/2431345111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0050199ae0 con 0x7f00501067f0 2023-07-14T17:36:37.704 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.381+0000 7f00477fe700 1 -- 172.21.15.88:0/2431345111 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f004c019640 con 0x7f00501067f0 2023-07-14T17:36:37.704 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.381+0000 7f0058676700 1 -- 172.21.15.88:0/2431345111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0050103db0 con 0x7f00501067f0 2023-07-14T17:36:37.704 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.381+0000 7f00477fe700 1 --2- 172.21.15.88:0/2431345111 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f003c038710 0x7f003c03abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:37.705 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.381+0000 7f00477fe700 1 -- 172.21.15.88:0/2431345111 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f004c04c6b0 con 0x7f00501067f0 2023-07-14T17:36:37.705 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.381+0000 7f0055c11700 1 --2- 172.21.15.88:0/2431345111 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f003c038710 0x7f003c03abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:37.705 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.382+0000 7f0055c11700 1 --2- 172.21.15.88:0/2431345111 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f003c038710 0x7f003c03abc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0040006fd0 tx=0x7f0040006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:37.705 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.386+0000 7f00477fe700 1 -- 172.21.15.88:0/2431345111 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f004c018350 con 0x7f00501067f0 2023-07-14T17:36:37.705 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:36.632+0000 7f0058676700 1 -- 172.21.15.88:0/2431345111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f0050103fa0 con 0x7f00501067f0 2023-07-14T17:36:37.705 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:37.186+0000 7f00477fe700 1 -- 172.21.15.88:0/2431345111 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f004c01a3d0 con 0x7f00501067f0 2023-07-14T17:36:37.705 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:37.188+0000 7f00477fe700 1 -- 172.21.15.88:0/2431345111 <== mon.0 v2:172.21.15.88:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f004c04e0c0 con 0x7f00501067f0 2023-07-14T17:36:37.705 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:37.191+0000 7f0058676700 1 -- 172.21.15.88:0/2431345111 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f003c038710 msgr2=0x7f003c03abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:37.706 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:37.191+0000 7f0058676700 1 --2- 172.21.15.88:0/2431345111 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f003c038710 0x7f003c03abc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0040006fd0 tx=0x7f0040006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:37.706 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:37.191+0000 7f0058676700 1 -- 172.21.15.88:0/2431345111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00501067f0 msgr2=0x7f0050198f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:37.706 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:37.191+0000 7f0058676700 1 --2- 172.21.15.88:0/2431345111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00501067f0 0x7f0050198f80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f004c006af0 tx=0x7f004c011e80 comp rx=0 tx=0).stop 2023-07-14T17:36:37.706 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:37.191+0000 7f0058676700 1 -- 172.21.15.88:0/2431345111 shutdown_connections 2023-07-14T17:36:37.706 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:37.191+0000 7f0058676700 1 --2- 172.21.15.88:0/2431345111 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f003c038710 0x7f003c03abc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:37.706 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:37.191+0000 7f0058676700 1 --2- 172.21.15.88:0/2431345111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00501067f0 0x7f0050198f80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:37.706 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:37.191+0000 7f0058676700 1 -- 172.21.15.88:0/2431345111 >> 172.21.15.88:0/2431345111 conn(0x7f00500feec0 msgr2=0x7f00500ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:37.707 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:37.191+0000 7f0058676700 1 -- 172.21.15.88:0/2431345111 shutdown_connections 2023-07-14T17:36:37.707 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:37.191+0000 7f0058676700 1 -- 172.21.15.88:0/2431345111 wait complete. 2023-07-14T17:36:37.810 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:37 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:37.634+0000 7f452d59d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T17:36:38.125 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:37 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:37.822+0000 7f452d59d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T17:36:38.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:38 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2431345111' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-14T17:36:38.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:38 smithi088 ceph-mon[127465]: mgrmap e9: x(active, since 16s) 2023-07-14T17:36:38.446 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:38 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:38.138+0000 7f452d59d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T17:36:38.447 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:38 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:38.232+0000 7f452d59d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T17:36:38.447 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:38 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:38.366+0000 7f452d59d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T17:36:38.882 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:38 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:38.461+0000 7f452d59d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T17:36:38.956 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout { 2023-07-14T17:36:38.957 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-07-14T17:36:38.957 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T17:36:38.957 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-14T17:36:38.957 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-14T17:36:38.957 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout } 2023-07-14T17:36:38.957 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.257+0000 7f45cef82700 1 Processor -- start 2023-07-14T17:36:38.958 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.257+0000 7f45cef82700 1 -- start start 2023-07-14T17:36:38.958 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.257+0000 7f45cef82700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c8105d30 0x7f45c8108110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:38.958 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.257+0000 7f45cef82700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c8079d10 con 0x7f45c8105d30 2023-07-14T17:36:38.958 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.257+0000 7f45ccd1e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c8105d30 0x7f45c8108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:38.958 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.257+0000 7f45ccd1e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c8105d30 0x7f45c8108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40346/0 (socket says 172.21.15.88:40346) 2023-07-14T17:36:38.958 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.257+0000 7f45ccd1e700 1 -- 172.21.15.88:0/1503552374 learned_addr learned my addr 172.21.15.88:0/1503552374 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:38.959 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.258+0000 7f45ccd1e700 1 -- 172.21.15.88:0/1503552374 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c8079e50 con 0x7f45c8105d30 2023-07-14T17:36:38.959 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.258+0000 7f45ccd1e700 1 --2- 172.21.15.88:0/1503552374 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c8105d30 0x7f45c8108110 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f45bc0098a0 tx=0x7f45bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e1ec13a11c53b5f server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:38.959 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.258+0000 7f45c77fe700 1 -- 172.21.15.88:0/1503552374 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45bc004160 con 0x7f45c8105d30 2023-07-14T17:36:38.959 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.258+0000 7f45c77fe700 1 -- 172.21.15.88:0/1503552374 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f45bc0042c0 con 0x7f45c8105d30 2023-07-14T17:36:38.959 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.258+0000 7f45c77fe700 1 -- 172.21.15.88:0/1503552374 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45bc0118e0 con 0x7f45c8105d30 2023-07-14T17:36:38.959 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.259+0000 7f45cef82700 1 -- 172.21.15.88:0/1503552374 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c8105d30 msgr2=0x7f45c8108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:38.960 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.259+0000 7f45cef82700 1 --2- 172.21.15.88:0/1503552374 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c8105d30 0x7f45c8108110 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f45bc0098a0 tx=0x7f45bc009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:38.960 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.259+0000 7f45cef82700 1 -- 172.21.15.88:0/1503552374 shutdown_connections 2023-07-14T17:36:38.960 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.259+0000 7f45cef82700 1 --2- 172.21.15.88:0/1503552374 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c8105d30 0x7f45c8108110 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:38.960 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.259+0000 7f45cef82700 1 -- 172.21.15.88:0/1503552374 >> 172.21.15.88:0/1503552374 conn(0x7f45c80ff770 msgr2=0x7f45c8101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:38.960 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.259+0000 7f45cef82700 1 -- 172.21.15.88:0/1503552374 shutdown_connections 2023-07-14T17:36:38.960 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.259+0000 7f45cef82700 1 -- 172.21.15.88:0/1503552374 wait complete. 2023-07-14T17:36:38.961 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.260+0000 7f45cef82700 1 Processor -- start 2023-07-14T17:36:38.961 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.260+0000 7f45cef82700 1 -- start start 2023-07-14T17:36:38.961 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.260+0000 7f45cef82700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c8105d30 0x7f45c8103ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:38.961 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.260+0000 7f45cef82700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c8105870 con 0x7f45c8105d30 2023-07-14T17:36:38.961 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.260+0000 7f45ccd1e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c8105d30 0x7f45c8103ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:38.961 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.260+0000 7f45ccd1e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c8105d30 0x7f45c8103ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40348/0 (socket says 172.21.15.88:40348) 2023-07-14T17:36:38.961 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.260+0000 7f45ccd1e700 1 -- 172.21.15.88:0/585868496 learned_addr learned my addr 172.21.15.88:0/585868496 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:38.962 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.260+0000 7f45ccd1e700 1 -- 172.21.15.88:0/585868496 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45bc009550 con 0x7f45c8105d30 2023-07-14T17:36:38.962 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.260+0000 7f45ccd1e700 1 --2- 172.21.15.88:0/585868496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c8105d30 0x7f45c8103ce0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f45bc000c00 tx=0x7f45bc0119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:38.962 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.261+0000 7f45c5ffb700 1 -- 172.21.15.88:0/585868496 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45bc0036a0 con 0x7f45c8105d30 2023-07-14T17:36:38.962 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.261+0000 7f45c5ffb700 1 -- 172.21.15.88:0/585868496 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f45bc01b440 con 0x7f45c8105d30 2023-07-14T17:36:38.962 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.261+0000 7f45cef82700 1 -- 172.21.15.88:0/585868496 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c8105a70 con 0x7f45c8105d30 2023-07-14T17:36:38.962 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.261+0000 7f45c5ffb700 1 -- 172.21.15.88:0/585868496 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45bc01a3d0 con 0x7f45c8105d30 2023-07-14T17:36:38.962 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.261+0000 7f45cef82700 1 -- 172.21.15.88:0/585868496 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c81043f0 con 0x7f45c8105d30 2023-07-14T17:36:38.963 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.262+0000 7f45c5ffb700 1 -- 172.21.15.88:0/585868496 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f45bc01a640 con 0x7f45c8105d30 2023-07-14T17:36:38.963 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.262+0000 7f45c5ffb700 1 --2- 172.21.15.88:0/585868496 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f45b80387b0 0x7f45b803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:38.963 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.262+0000 7f45c5ffb700 1 -- 172.21.15.88:0/585868496 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f45bc04c610 con 0x7f45c8105d30 2023-07-14T17:36:38.963 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.262+0000 7f45c7fff700 1 -- 172.21.15.88:0/585868496 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f45b80387b0 msgr2=0x7f45b803ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.88:6800/2424184187 2023-07-14T17:36:38.963 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.262+0000 7f45c7fff700 1 --2- 172.21.15.88:0/585868496 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f45b80387b0 0x7f45b803ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T17:36:38.963 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.262+0000 7f45cef82700 1 -- 172.21.15.88:0/585868496 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45c8068520 con 0x7f45c8105d30 2023-07-14T17:36:38.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.267+0000 7f45c5ffb700 1 -- 172.21.15.88:0/585868496 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f45bc01f020 con 0x7f45c8105d30 2023-07-14T17:36:38.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.463+0000 7f45c7fff700 1 -- 172.21.15.88:0/585868496 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f45b80387b0 msgr2=0x7f45b803ac60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.88:6800/2424184187 2023-07-14T17:36:38.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.463+0000 7f45c7fff700 1 --2- 172.21.15.88:0/585868496 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f45b80387b0 0x7f45b803ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T17:36:38.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.511+0000 7f45cef82700 1 -- 172.21.15.88:0/585868496 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f45c8104630 con 0x7f45c8105d30 2023-07-14T17:36:38.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.511+0000 7f45c5ffb700 1 -- 172.21.15.88:0/585868496 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f45bc01f070 con 0x7f45c8105d30 2023-07-14T17:36:38.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.514+0000 7f45cef82700 1 -- 172.21.15.88:0/585868496 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f45b80387b0 msgr2=0x7f45b803ac60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T17:36:38.964 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.514+0000 7f45cef82700 1 --2- 172.21.15.88:0/585868496 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f45b80387b0 0x7f45b803ac60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:38.965 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.514+0000 7f45cef82700 1 -- 172.21.15.88:0/585868496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c8105d30 msgr2=0x7f45c8103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:38.965 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.514+0000 7f45cef82700 1 --2- 172.21.15.88:0/585868496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c8105d30 0x7f45c8103ce0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f45bc000c00 tx=0x7f45bc0119a0 comp rx=0 tx=0).stop 2023-07-14T17:36:38.965 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.514+0000 7f45cef82700 1 -- 172.21.15.88:0/585868496 shutdown_connections 2023-07-14T17:36:38.965 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.514+0000 7f45cef82700 1 --2- 172.21.15.88:0/585868496 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f45b80387b0 0x7f45b803ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:38.965 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.514+0000 7f45cef82700 1 --2- 172.21.15.88:0/585868496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f45c8105d30 0x7f45c8103ce0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:38.965 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.514+0000 7f45cef82700 1 -- 172.21.15.88:0/585868496 >> 172.21.15.88:0/585868496 conn(0x7f45c80ff770 msgr2=0x7f45c8100e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:38.965 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.514+0000 7f45cef82700 1 -- 172.21.15.88:0/585868496 shutdown_connections 2023-07-14T17:36:38.966 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:38.514+0000 7f45cef82700 1 -- 172.21.15.88:0/585868496 wait complete. 2023-07-14T17:36:38.966 INFO:teuthology.orchestra.run.smithi088.stdout:Waiting for the mgr to restart... 2023-07-14T17:36:38.966 INFO:teuthology.orchestra.run.smithi088.stdout:Waiting for mgr epoch 9... 2023-07-14T17:36:39.161 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:38 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:38.894+0000 7f452d59d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T17:36:39.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:39 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/585868496' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-14T17:36:39.946 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:39 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:39.485+0000 7f452d59d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T17:36:39.946 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:39 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:39.653+0000 7f452d59d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T17:36:40.658 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:40 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:40.409+0000 7f452d59d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T17:36:40.658 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:40 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:40.499+0000 7f452d59d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T17:36:40.937 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:40 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:40.670+0000 7f452d59d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T17:36:41.196 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:40 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:40.950+0000 7f452d59d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T17:36:41.196 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:41 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:41.177+0000 7f452d59d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T17:36:41.572 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:41 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:41.325+0000 7f452d59d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T17:36:41.946 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:41 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:41.584+0000 7f452d59d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T17:36:42.591 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:42 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:42.279+0000 7f452d59d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T17:36:42.592 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:42 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:42.412+0000 7f452d59d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T17:36:42.923 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:42 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:42.603+0000 7f452d59d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T17:36:42.924 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:42 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:42.718+0000 7f452d59d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T17:36:43.196 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:42 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:42.936+0000 7f452d59d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T17:36:44.196 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:36:43 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:36:43.785+0000 7f452d59d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T17:36:44.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:43 smithi088 ceph-mon[127465]: Active manager daemon x restarted 2023-07-14T17:36:44.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:43 smithi088 ceph-mon[127465]: Activating manager daemon x 2023-07-14T17:36:44.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:43 smithi088 ceph-mon[127465]: osdmap e3: 0 total, 0 up, 0 in 2023-07-14T17:36:44.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:43 smithi088 ceph-mon[127465]: mgrmap e10: x(active, starting, since 0.00309058s) 2023-07-14T17:36:44.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T17:36:44.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T17:36:44.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T17:36:44.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T17:36:44.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T17:36:44.198 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:43 smithi088 ceph-mon[127465]: Manager daemon x is now available 2023-07-14T17:36:44.198 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:36:44.198 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:36:45.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T17:36:45.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T17:36:45.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:44 smithi088 ceph-mon[127465]: [14/Jul/2023:17:36:44] ENGINE Bus STARTING 2023-07-14T17:36:45.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:44 smithi088 ceph-mon[127465]: mgrmap e11: x(active, since 1.00558s) 2023-07-14T17:36:45.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout { 2023-07-14T17:36:45.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-07-14T17:36:45.415 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-14T17:36:45.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout } 2023-07-14T17:36:45.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.467+0000 7f482fa43700 1 Processor -- start 2023-07-14T17:36:45.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.467+0000 7f482fa43700 1 -- start start 2023-07-14T17:36:45.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.468+0000 7f482fa43700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48281040e0 0x7f48281044f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:45.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.468+0000 7f482fa43700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4828104a30 con 0x7f48281040e0 2023-07-14T17:36:45.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.468+0000 7f482d7df700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48281040e0 0x7f48281044f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:45.416 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.468+0000 7f482d7df700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48281040e0 0x7f48281044f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40356/0 (socket says 172.21.15.88:40356) 2023-07-14T17:36:45.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.468+0000 7f482d7df700 1 -- 172.21.15.88:0/2013180312 learned_addr learned my addr 172.21.15.88:0/2013180312 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:45.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.468+0000 7f482d7df700 1 -- 172.21.15.88:0/2013180312 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4828104b70 con 0x7f48281040e0 2023-07-14T17:36:45.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.468+0000 7f482d7df700 1 --2- 172.21.15.88:0/2013180312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48281040e0 0x7f48281044f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4820009e80 tx=0x7f482000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d854f22603ccb7 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:45.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.468+0000 7f481ffff700 1 -- 172.21.15.88:0/2013180312 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f482000e070 con 0x7f48281040e0 2023-07-14T17:36:45.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.468+0000 7f481ffff700 1 -- 172.21.15.88:0/2013180312 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f482000b8c0 con 0x7f48281040e0 2023-07-14T17:36:45.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.468+0000 7f481ffff700 1 -- 172.21.15.88:0/2013180312 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4820004360 con 0x7f48281040e0 2023-07-14T17:36:45.417 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.469+0000 7f482fa43700 1 -- 172.21.15.88:0/2013180312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48281040e0 msgr2=0x7f48281044f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:45.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.469+0000 7f482fa43700 1 --2- 172.21.15.88:0/2013180312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48281040e0 0x7f48281044f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4820009e80 tx=0x7f482000b330 comp rx=0 tx=0).stop 2023-07-14T17:36:45.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.469+0000 7f482fa43700 1 -- 172.21.15.88:0/2013180312 shutdown_connections 2023-07-14T17:36:45.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.469+0000 7f482fa43700 1 --2- 172.21.15.88:0/2013180312 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48281040e0 0x7f48281044f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:45.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.469+0000 7f482fa43700 1 -- 172.21.15.88:0/2013180312 >> 172.21.15.88:0/2013180312 conn(0x7f48280ff770 msgr2=0x7f4828101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:45.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.469+0000 7f482fa43700 1 -- 172.21.15.88:0/2013180312 shutdown_connections 2023-07-14T17:36:45.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.469+0000 7f482fa43700 1 -- 172.21.15.88:0/2013180312 wait complete. 2023-07-14T17:36:45.418 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.470+0000 7f482fa43700 1 Processor -- start 2023-07-14T17:36:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.470+0000 7f482fa43700 1 -- start start 2023-07-14T17:36:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.470+0000 7f482fa43700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48281040e0 0x7f482819bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.470+0000 7f482fa43700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4828079ce0 con 0x7f48281040e0 2023-07-14T17:36:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.470+0000 7f482d7df700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48281040e0 0x7f482819bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.470+0000 7f482d7df700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48281040e0 0x7f482819bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40358/0 (socket says 172.21.15.88:40358) 2023-07-14T17:36:45.419 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.470+0000 7f482d7df700 1 -- 172.21.15.88:0/2091217384 learned_addr learned my addr 172.21.15.88:0/2091217384 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.470+0000 7f482d7df700 1 -- 172.21.15.88:0/2091217384 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4820009b30 con 0x7f48281040e0 2023-07-14T17:36:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.470+0000 7f482d7df700 1 --2- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48281040e0 0x7f482819bc40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4820009530 tx=0x7f4820003c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.471+0000 7f481e7fc700 1 -- 172.21.15.88:0/2091217384 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f482000e040 con 0x7f48281040e0 2023-07-14T17:36:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.471+0000 7f481e7fc700 1 -- 172.21.15.88:0/2091217384 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f482000be30 con 0x7f48281040e0 2023-07-14T17:36:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.471+0000 7f482fa43700 1 -- 172.21.15.88:0/2091217384 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4828079ee0 con 0x7f48281040e0 2023-07-14T17:36:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.471+0000 7f481e7fc700 1 -- 172.21.15.88:0/2091217384 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f482001c570 con 0x7f48281040e0 2023-07-14T17:36:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.471+0000 7f482fa43700 1 -- 172.21.15.88:0/2091217384 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f482819c3b0 con 0x7f48281040e0 2023-07-14T17:36:45.420 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.472+0000 7f481e7fc700 1 -- 172.21.15.88:0/2091217384 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f4820012430 con 0x7f48281040e0 2023-07-14T17:36:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.472+0000 7f481e7fc700 1 --2- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4814038760 0x7f481403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.472+0000 7f481e7fc700 1 -- 172.21.15.88:0/2091217384 --> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f481403b320 con 0x7f4814038760 2023-07-14T17:36:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.472+0000 7f481e7fc700 1 -- 172.21.15.88:0/2091217384 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f482004c6d0 con 0x7f48281040e0 2023-07-14T17:36:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.472+0000 7f482cfde700 1 -- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4814038760 msgr2=0x7f481403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.88:6800/2424184187 2023-07-14T17:36:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.472+0000 7f482cfde700 1 --2- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4814038760 0x7f481403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T17:36:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.673+0000 7f482cfde700 1 -- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4814038760 msgr2=0x7f481403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.88:6800/2424184187 2023-07-14T17:36:45.421 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:39.673+0000 7f482cfde700 1 --2- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4814038760 0x7f481403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T17:36:45.422 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:40.074+0000 7f482cfde700 1 -- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4814038760 msgr2=0x7f481403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.88:6800/2424184187 2023-07-14T17:36:45.422 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:40.074+0000 7f482cfde700 1 --2- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4814038760 0x7f481403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-14T17:36:45.422 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:40.875+0000 7f482cfde700 1 -- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4814038760 msgr2=0x7f481403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.88:6800/2424184187 2023-07-14T17:36:45.422 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:40.875+0000 7f482cfde700 1 --2- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4814038760 0x7f481403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-14T17:36:45.422 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:42.476+0000 7f482cfde700 1 -- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4814038760 msgr2=0x7f481403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.88:6800/2424184187 2023-07-14T17:36:45.422 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:42.476+0000 7f482cfde700 1 --2- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4814038760 0x7f481403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-14T17:36:45.422 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:43.800+0000 7f481e7fc700 1 -- 172.21.15.88:0/2091217384 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7f482002d350 con 0x7f48281040e0 2023-07-14T17:36:45.423 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:43.800+0000 7f481e7fc700 1 -- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4814038760 msgr2=0x7f481403ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T17:36:45.423 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:43.800+0000 7f481e7fc700 1 --2- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4814038760 0x7f481403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:45.423 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.801+0000 7f481e7fc700 1 -- 172.21.15.88:0/2091217384 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f482004e4a0 con 0x7f48281040e0 2023-07-14T17:36:45.423 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.801+0000 7f481e7fc700 1 --2- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f481403bff0 0x7f481403e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:45.423 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.802+0000 7f481e7fc700 1 -- 172.21.15.88:0/2091217384 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f481403b320 con 0x7f481403bff0 2023-07-14T17:36:45.423 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.802+0000 7f482cfde700 1 --2- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f481403bff0 0x7f481403e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:45.423 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.802+0000 7f482cfde700 1 --2- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f481403bff0 0x7f481403e3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4818003a10 tx=0x7f4818009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:45.423 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.803+0000 7f481e7fc700 1 -- 172.21.15.88:0/2091217384 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f481403b320 con 0x7f481403bff0 2023-07-14T17:36:45.424 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.809+0000 7f482fa43700 1 -- 172.21.15.88:0/2091217384 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f48280796a0 con 0x7f481403bff0 2023-07-14T17:36:45.424 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.809+0000 7f481e7fc700 1 -- 172.21.15.88:0/2091217384 <== mgr.14152 v2:172.21.15.88:6800/4262759610 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f48280796a0 con 0x7f481403bff0 2023-07-14T17:36:45.424 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.810+0000 7f482fa43700 1 -- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f481403bff0 msgr2=0x7f481403e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:45.424 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.810+0000 7f482fa43700 1 --2- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f481403bff0 0x7f481403e3d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4818003a10 tx=0x7f4818009270 comp rx=0 tx=0).stop 2023-07-14T17:36:45.424 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.810+0000 7f482fa43700 1 -- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48281040e0 msgr2=0x7f482819bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:45.424 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.810+0000 7f482fa43700 1 --2- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48281040e0 0x7f482819bc40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4820009530 tx=0x7f4820003c50 comp rx=0 tx=0).stop 2023-07-14T17:36:45.424 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.810+0000 7f482fa43700 1 -- 172.21.15.88:0/2091217384 shutdown_connections 2023-07-14T17:36:45.425 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.810+0000 7f482fa43700 1 --2- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f481403bff0 0x7f481403e3d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:45.425 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.810+0000 7f482fa43700 1 --2- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48281040e0 0x7f482819bc40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:45.425 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.810+0000 7f482fa43700 1 --2- 172.21.15.88:0/2091217384 >> [v2:172.21.15.88:6800/2424184187,v1:172.21.15.88:6801/2424184187] conn(0x7f4814038760 0x7f481403ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:45.425 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.810+0000 7f482fa43700 1 -- 172.21.15.88:0/2091217384 >> 172.21.15.88:0/2091217384 conn(0x7f48280ff770 msgr2=0x7f4828100500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:45.425 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.810+0000 7f482fa43700 1 -- 172.21.15.88:0/2091217384 shutdown_connections 2023-07-14T17:36:45.425 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:44.810+0000 7f482fa43700 1 -- 172.21.15.88:0/2091217384 wait complete. 2023-07-14T17:36:45.425 INFO:teuthology.orchestra.run.smithi088.stdout:mgr epoch 9 is available 2023-07-14T17:36:45.426 INFO:teuthology.orchestra.run.smithi088.stdout:Generating a dashboard self-signed certificate... 2023-07-14T17:36:45.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:45 smithi088 ceph-mon[127465]: [14/Jul/2023:17:36:44] ENGINE Serving on http://172.21.15.88:8765 2023-07-14T17:36:45.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:45 smithi088 ceph-mon[127465]: [14/Jul/2023:17:36:44] ENGINE Serving on https://172.21.15.88:7150 2023-07-14T17:36:45.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:45 smithi088 ceph-mon[127465]: [14/Jul/2023:17:36:44] ENGINE Bus STARTED 2023-07-14T17:36:45.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:45 smithi088 ceph-mon[127465]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-14T17:36:45.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:45 smithi088 ceph-mon[127465]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-14T17:36:45.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:45 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:36:46.553 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-07-14T17:36:46.554 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.931+0000 7fb473313700 1 Processor -- start 2023-07-14T17:36:46.554 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.931+0000 7fb473313700 1 -- start start 2023-07-14T17:36:46.554 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.931+0000 7fb473313700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb46c105d30 0x7fb46c108110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:46.554 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.931+0000 7fb473313700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c079d10 con 0x7fb46c105d30 2023-07-14T17:36:46.554 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.932+0000 7fb4710af700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb46c105d30 0x7fb46c108110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:46.555 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.932+0000 7fb4710af700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb46c105d30 0x7fb46c108110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40412/0 (socket says 172.21.15.88:40412) 2023-07-14T17:36:46.555 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.932+0000 7fb4710af700 1 -- 172.21.15.88:0/3629433678 learned_addr learned my addr 172.21.15.88:0/3629433678 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:46.555 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.932+0000 7fb4710af700 1 -- 172.21.15.88:0/3629433678 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46c079e50 con 0x7fb46c105d30 2023-07-14T17:36:46.555 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.932+0000 7fb4710af700 1 --2- 172.21.15.88:0/3629433678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb46c105d30 0x7fb46c108110 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb4680098a0 tx=0x7fb468009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179d0cb5b0e7287c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:46.555 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.932+0000 7fb463fff700 1 -- 172.21.15.88:0/3629433678 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb468004160 con 0x7fb46c105d30 2023-07-14T17:36:46.555 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.932+0000 7fb463fff700 1 -- 172.21.15.88:0/3629433678 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb4680042c0 con 0x7fb46c105d30 2023-07-14T17:36:46.556 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.932+0000 7fb463fff700 1 -- 172.21.15.88:0/3629433678 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4680118e0 con 0x7fb46c105d30 2023-07-14T17:36:46.556 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.933+0000 7fb473313700 1 -- 172.21.15.88:0/3629433678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb46c105d30 msgr2=0x7fb46c108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:46.556 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.933+0000 7fb473313700 1 --2- 172.21.15.88:0/3629433678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb46c105d30 0x7fb46c108110 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb4680098a0 tx=0x7fb468009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:46.556 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.933+0000 7fb473313700 1 -- 172.21.15.88:0/3629433678 shutdown_connections 2023-07-14T17:36:46.556 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.933+0000 7fb473313700 1 --2- 172.21.15.88:0/3629433678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb46c105d30 0x7fb46c108110 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:46.557 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.933+0000 7fb473313700 1 -- 172.21.15.88:0/3629433678 >> 172.21.15.88:0/3629433678 conn(0x7fb46c0ff770 msgr2=0x7fb46c101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:46.558 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.933+0000 7fb473313700 1 -- 172.21.15.88:0/3629433678 shutdown_connections 2023-07-14T17:36:46.558 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.933+0000 7fb473313700 1 -- 172.21.15.88:0/3629433678 wait complete. 2023-07-14T17:36:46.558 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.933+0000 7fb473313700 1 Processor -- start 2023-07-14T17:36:46.558 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.933+0000 7fb473313700 1 -- start start 2023-07-14T17:36:46.558 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.934+0000 7fb473313700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb46c105d30 0x7fb46c103ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:46.558 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.934+0000 7fb473313700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c105870 con 0x7fb46c105d30 2023-07-14T17:36:46.559 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.934+0000 7fb4710af700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb46c105d30 0x7fb46c103ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:46.559 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.934+0000 7fb4710af700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb46c105d30 0x7fb46c103ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40414/0 (socket says 172.21.15.88:40414) 2023-07-14T17:36:46.559 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.934+0000 7fb4710af700 1 -- 172.21.15.88:0/1722593525 learned_addr learned my addr 172.21.15.88:0/1722593525 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:46.559 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.934+0000 7fb4710af700 1 -- 172.21.15.88:0/1722593525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb468009550 con 0x7fb46c105d30 2023-07-14T17:36:46.559 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.934+0000 7fb4710af700 1 --2- 172.21.15.88:0/1722593525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb46c105d30 0x7fb46c103ce0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb468006af0 tx=0x7fb468011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:46.559 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.934+0000 7fb4627fc700 1 -- 172.21.15.88:0/1722593525 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4680037d0 con 0x7fb46c105d30 2023-07-14T17:36:46.559 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.934+0000 7fb4627fc700 1 -- 172.21.15.88:0/1722593525 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb46801b440 con 0x7fb46c105d30 2023-07-14T17:36:46.560 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.934+0000 7fb4627fc700 1 -- 172.21.15.88:0/1722593525 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb468019560 con 0x7fb46c105d30 2023-07-14T17:36:46.560 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.935+0000 7fb473313700 1 -- 172.21.15.88:0/1722593525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46c105ad0 con 0x7fb46c105d30 2023-07-14T17:36:46.560 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.935+0000 7fb473313700 1 -- 172.21.15.88:0/1722593525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46c1043f0 con 0x7fb46c105d30 2023-07-14T17:36:46.560 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.936+0000 7fb4627fc700 1 -- 172.21.15.88:0/1722593525 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7fb468003930 con 0x7fb46c105d30 2023-07-14T17:36:46.560 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.936+0000 7fb473313700 1 -- 172.21.15.88:0/1722593525 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb46c068520 con 0x7fb46c105d30 2023-07-14T17:36:46.560 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.936+0000 7fb4627fc700 1 --2- 172.21.15.88:0/1722593525 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb458038640 0x7fb45803aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:46.560 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.936+0000 7fb4627fc700 1 -- 172.21.15.88:0/1722593525 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb46804c490 con 0x7fb46c105d30 2023-07-14T17:36:46.561 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.936+0000 7fb4708ae700 1 --2- 172.21.15.88:0/1722593525 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb458038640 0x7fb45803aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:46.561 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.936+0000 7fb4708ae700 1 --2- 172.21.15.88:0/1722593525 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb458038640 0x7fb45803aaf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb45c006fd0 tx=0x7fb45c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:46.561 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:45.941+0000 7fb4627fc700 1 -- 172.21.15.88:0/1722593525 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb46801b5b0 con 0x7fb46c105d30 2023-07-14T17:36:46.561 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:46.142+0000 7fb473313700 1 -- 172.21.15.88:0/1722593525 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fb46c104580 con 0x7fb458038640 2023-07-14T17:36:46.561 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:46.211+0000 7fb4627fc700 1 -- 172.21.15.88:0/1722593525 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fb46c104580 con 0x7fb458038640 2023-07-14T17:36:46.561 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:46.213+0000 7fb473313700 1 -- 172.21.15.88:0/1722593525 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb458038640 msgr2=0x7fb45803aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:46.561 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:46.213+0000 7fb473313700 1 --2- 172.21.15.88:0/1722593525 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb458038640 0x7fb45803aaf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb45c006fd0 tx=0x7fb45c006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:46.562 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:46.213+0000 7fb473313700 1 -- 172.21.15.88:0/1722593525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb46c105d30 msgr2=0x7fb46c103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:46.562 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:46.213+0000 7fb473313700 1 --2- 172.21.15.88:0/1722593525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb46c105d30 0x7fb46c103ce0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb468006af0 tx=0x7fb468011e80 comp rx=0 tx=0).stop 2023-07-14T17:36:46.562 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:46.213+0000 7fb473313700 1 -- 172.21.15.88:0/1722593525 shutdown_connections 2023-07-14T17:36:46.562 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:46.213+0000 7fb473313700 1 --2- 172.21.15.88:0/1722593525 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb458038640 0x7fb45803aaf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:46.562 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:46.213+0000 7fb473313700 1 --2- 172.21.15.88:0/1722593525 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb46c105d30 0x7fb46c103ce0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:46.562 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:46.213+0000 7fb473313700 1 -- 172.21.15.88:0/1722593525 >> 172.21.15.88:0/1722593525 conn(0x7fb46c0ff770 msgr2=0x7fb46c100e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:46.562 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:46.213+0000 7fb473313700 1 -- 172.21.15.88:0/1722593525 shutdown_connections 2023-07-14T17:36:46.563 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:46.213+0000 7fb473313700 1 -- 172.21.15.88:0/1722593525 wait complete. 2023-07-14T17:36:46.563 INFO:teuthology.orchestra.run.smithi088.stdout:Creating initial admin user... 2023-07-14T17:36:47.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:47 smithi088 ceph-mon[127465]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:36:47.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:47 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:36:47.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:47 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:36:47.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:47 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:36:47.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:47 smithi088 ceph-mon[127465]: mgrmap e12: x(active, since 2s) 2023-07-14T17:36:48.003 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$fzRouijHPKwdTk1C9HH3.ebI1fS4zuaNKSRB9nMk0fjHvYfe9Nfpq", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1689356207, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-14T17:36:48.004 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.027+0000 7f27fe4e0700 1 Processor -- start 2023-07-14T17:36:48.004 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.027+0000 7f27fe4e0700 1 -- start start 2023-07-14T17:36:48.004 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.027+0000 7f27fe4e0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27f8105d60 0x7f27f8108140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:48.004 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.027+0000 7f27fe4e0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f8079cf0 con 0x7f27f8105d60 2023-07-14T17:36:48.004 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.027+0000 7f27f7fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27f8105d60 0x7f27f8108140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:48.004 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.027+0000 7f27f7fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27f8105d60 0x7f27f8108140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40418/0 (socket says 172.21.15.88:40418) 2023-07-14T17:36:48.004 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.027+0000 7f27f7fff700 1 -- 172.21.15.88:0/2781209948 learned_addr learned my addr 172.21.15.88:0/2781209948 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:48.005 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.027+0000 7f27f7fff700 1 -- 172.21.15.88:0/2781209948 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f8079e30 con 0x7f27f8105d60 2023-07-14T17:36:48.005 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.028+0000 7f27f7fff700 1 --2- 172.21.15.88:0/2781209948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27f8105d60 0x7f27f8108140 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f27e80098a0 tx=0x7f27e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c336ea5887ce5fd0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:48.005 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.028+0000 7f27f6ffd700 1 -- 172.21.15.88:0/2781209948 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27e8004160 con 0x7f27f8105d60 2023-07-14T17:36:48.005 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.028+0000 7f27f6ffd700 1 -- 172.21.15.88:0/2781209948 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f27e80042c0 con 0x7f27f8105d60 2023-07-14T17:36:48.005 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.028+0000 7f27f6ffd700 1 -- 172.21.15.88:0/2781209948 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27e80118e0 con 0x7f27f8105d60 2023-07-14T17:36:48.005 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.028+0000 7f27fe4e0700 1 -- 172.21.15.88:0/2781209948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27f8105d60 msgr2=0x7f27f8108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:48.005 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.028+0000 7f27fe4e0700 1 --2- 172.21.15.88:0/2781209948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27f8105d60 0x7f27f8108140 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f27e80098a0 tx=0x7f27e8009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:48.006 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.028+0000 7f27fe4e0700 1 -- 172.21.15.88:0/2781209948 shutdown_connections 2023-07-14T17:36:48.006 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.028+0000 7f27fe4e0700 1 --2- 172.21.15.88:0/2781209948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27f8105d60 0x7f27f8108140 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:48.006 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.028+0000 7f27fe4e0700 1 -- 172.21.15.88:0/2781209948 >> 172.21.15.88:0/2781209948 conn(0x7f27f80ff7a0 msgr2=0x7f27f8101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:48.006 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.028+0000 7f27fe4e0700 1 -- 172.21.15.88:0/2781209948 shutdown_connections 2023-07-14T17:36:48.006 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.028+0000 7f27fe4e0700 1 -- 172.21.15.88:0/2781209948 wait complete. 2023-07-14T17:36:48.006 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.029+0000 7f27fe4e0700 1 Processor -- start 2023-07-14T17:36:48.006 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.029+0000 7f27fe4e0700 1 -- start start 2023-07-14T17:36:48.007 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.029+0000 7f27fe4e0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27f8105d60 0x7f27f8199040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:48.007 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.029+0000 7f27fe4e0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f8199580 con 0x7f27f8105d60 2023-07-14T17:36:48.007 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.029+0000 7f27f7fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27f8105d60 0x7f27f8199040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:48.007 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.029+0000 7f27f7fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27f8105d60 0x7f27f8199040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40420/0 (socket says 172.21.15.88:40420) 2023-07-14T17:36:48.007 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.029+0000 7f27f7fff700 1 -- 172.21.15.88:0/1109157649 learned_addr learned my addr 172.21.15.88:0/1109157649 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:48.007 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.030+0000 7f27f7fff700 1 -- 172.21.15.88:0/1109157649 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e8009550 con 0x7f27f8105d60 2023-07-14T17:36:48.007 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.030+0000 7f27f7fff700 1 --2- 172.21.15.88:0/1109157649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27f8105d60 0x7f27f8199040 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f27e8006af0 tx=0x7f27e8011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:48.007 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.030+0000 7f27f57fa700 1 -- 172.21.15.88:0/1109157649 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27e80037d0 con 0x7f27f8105d60 2023-07-14T17:36:48.008 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.030+0000 7f27f57fa700 1 -- 172.21.15.88:0/1109157649 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f27e801b440 con 0x7f27f8105d60 2023-07-14T17:36:48.008 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.030+0000 7f27fe4e0700 1 -- 172.21.15.88:0/1109157649 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f8199780 con 0x7f27f8105d60 2023-07-14T17:36:48.008 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.030+0000 7f27f57fa700 1 -- 172.21.15.88:0/1109157649 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27e8019560 con 0x7f27f8105d60 2023-07-14T17:36:48.008 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.030+0000 7f27fe4e0700 1 -- 172.21.15.88:0/1109157649 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f8199ba0 con 0x7f27f8105d60 2023-07-14T17:36:48.008 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.031+0000 7f27f57fa700 1 -- 172.21.15.88:0/1109157649 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f27e8003930 con 0x7f27f8105d60 2023-07-14T17:36:48.008 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.031+0000 7f27fe4e0700 1 -- 172.21.15.88:0/1109157649 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f81092d0 con 0x7f27f8105d60 2023-07-14T17:36:48.008 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.031+0000 7f27f57fa700 1 --2- 172.21.15.88:0/1109157649 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f27e0038710 0x7f27e003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:48.009 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.031+0000 7f27f57fa700 1 -- 172.21.15.88:0/1109157649 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f27e804c700 con 0x7f27f8105d60 2023-07-14T17:36:48.009 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.031+0000 7f27f77fe700 1 --2- 172.21.15.88:0/1109157649 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f27e0038710 0x7f27e003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:48.009 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.032+0000 7f27f77fe700 1 --2- 172.21.15.88:0/1109157649 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f27e0038710 0x7f27e003abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f27ec006fd0 tx=0x7f27ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:48.009 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.036+0000 7f27f57fa700 1 -- 172.21.15.88:0/1109157649 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f27e802bb40 con 0x7f27f8105d60 2023-07-14T17:36:48.009 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.264+0000 7f27fe4e0700 1 -- 172.21.15.88:0/1109157649 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f27f8061a90 con 0x7f27e0038710 2023-07-14T17:36:48.009 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.489+0000 7f27f57fa700 1 -- 172.21.15.88:0/1109157649 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f27f8061a90 con 0x7f27e0038710 2023-07-14T17:36:48.009 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.491+0000 7f27fe4e0700 1 -- 172.21.15.88:0/1109157649 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f27e0038710 msgr2=0x7f27e003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:48.010 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.491+0000 7f27fe4e0700 1 --2- 172.21.15.88:0/1109157649 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f27e0038710 0x7f27e003abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f27ec006fd0 tx=0x7f27ec006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:48.010 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.492+0000 7f27fe4e0700 1 -- 172.21.15.88:0/1109157649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27f8105d60 msgr2=0x7f27f8199040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:48.010 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.492+0000 7f27fe4e0700 1 --2- 172.21.15.88:0/1109157649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27f8105d60 0x7f27f8199040 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f27e8006af0 tx=0x7f27e8011e80 comp rx=0 tx=0).stop 2023-07-14T17:36:48.010 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.492+0000 7f27fe4e0700 1 -- 172.21.15.88:0/1109157649 shutdown_connections 2023-07-14T17:36:48.010 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.492+0000 7f27fe4e0700 1 --2- 172.21.15.88:0/1109157649 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f27e0038710 0x7f27e003abc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:48.010 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.492+0000 7f27fe4e0700 1 --2- 172.21.15.88:0/1109157649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27f8105d60 0x7f27f8199040 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:48.010 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.492+0000 7f27fe4e0700 1 -- 172.21.15.88:0/1109157649 >> 172.21.15.88:0/1109157649 conn(0x7f27f80ff7a0 msgr2=0x7f27f8100eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:48.010 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.492+0000 7f27fe4e0700 1 -- 172.21.15.88:0/1109157649 shutdown_connections 2023-07-14T17:36:48.011 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:47.492+0000 7f27fe4e0700 1 -- 172.21.15.88:0/1109157649 wait complete. 2023-07-14T17:36:48.011 INFO:teuthology.orchestra.run.smithi088.stdout:Fetching dashboard port number... 2023-07-14T17:36:48.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:48 smithi088 ceph-mon[127465]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:36:48.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:48 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:36:49.159 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stdout 8443 2023-07-14T17:36:49.159 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.621+0000 7f69cf144700 1 Processor -- start 2023-07-14T17:36:49.160 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.621+0000 7f69cf144700 1 -- start start 2023-07-14T17:36:49.160 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.622+0000 7f69cf144700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69c8106890 0x7f69c8106ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:49.160 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.622+0000 7f69cf144700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c81071e0 con 0x7f69c8106890 2023-07-14T17:36:49.160 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.622+0000 7f69ccee0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69c8106890 0x7f69c8106ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:49.161 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.622+0000 7f69ccee0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69c8106890 0x7f69c8106ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40424/0 (socket says 172.21.15.88:40424) 2023-07-14T17:36:49.161 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.622+0000 7f69ccee0700 1 -- 172.21.15.88:0/3216381753 learned_addr learned my addr 172.21.15.88:0/3216381753 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:49.161 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.622+0000 7f69ccee0700 1 -- 172.21.15.88:0/3216381753 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c8107320 con 0x7f69c8106890 2023-07-14T17:36:49.162 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.622+0000 7f69ccee0700 1 --2- 172.21.15.88:0/3216381753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69c8106890 0x7f69c8106ca0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f69bc0098a0 tx=0x7f69bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d06d6c393e05d9c2 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:49.162 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.623+0000 7f69c77fe700 1 -- 172.21.15.88:0/3216381753 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69bc004160 con 0x7f69c8106890 2023-07-14T17:36:49.162 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.623+0000 7f69c77fe700 1 -- 172.21.15.88:0/3216381753 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f69bc0042c0 con 0x7f69c8106890 2023-07-14T17:36:49.162 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.623+0000 7f69c77fe700 1 -- 172.21.15.88:0/3216381753 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69bc0118e0 con 0x7f69c8106890 2023-07-14T17:36:49.163 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.623+0000 7f69cf144700 1 -- 172.21.15.88:0/3216381753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69c8106890 msgr2=0x7f69c8106ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:49.163 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.623+0000 7f69cf144700 1 --2- 172.21.15.88:0/3216381753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69c8106890 0x7f69c8106ca0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f69bc0098a0 tx=0x7f69bc009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:49.163 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.623+0000 7f69cf144700 1 -- 172.21.15.88:0/3216381753 shutdown_connections 2023-07-14T17:36:49.163 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.623+0000 7f69cf144700 1 --2- 172.21.15.88:0/3216381753 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69c8106890 0x7f69c8106ca0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:49.163 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.623+0000 7f69cf144700 1 -- 172.21.15.88:0/3216381753 >> 172.21.15.88:0/3216381753 conn(0x7f69c80feec0 msgr2=0x7f69c81012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:49.163 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.623+0000 7f69cf144700 1 -- 172.21.15.88:0/3216381753 shutdown_connections 2023-07-14T17:36:49.163 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.623+0000 7f69cf144700 1 -- 172.21.15.88:0/3216381753 wait complete. 2023-07-14T17:36:49.164 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.624+0000 7f69cf144700 1 Processor -- start 2023-07-14T17:36:49.164 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.624+0000 7f69cf144700 1 -- start start 2023-07-14T17:36:49.164 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.624+0000 7f69cf144700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69c8106890 0x7f69c81045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:49.164 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.624+0000 7f69cf144700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c8106170 con 0x7f69c8106890 2023-07-14T17:36:49.164 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.624+0000 7f69ccee0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69c8106890 0x7f69c81045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:49.164 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.624+0000 7f69ccee0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69c8106890 0x7f69c81045e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40426/0 (socket says 172.21.15.88:40426) 2023-07-14T17:36:49.164 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.624+0000 7f69ccee0700 1 -- 172.21.15.88:0/2575512088 learned_addr learned my addr 172.21.15.88:0/2575512088 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:49.165 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.624+0000 7f69ccee0700 1 -- 172.21.15.88:0/2575512088 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69bc009550 con 0x7f69c8106890 2023-07-14T17:36:49.165 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.624+0000 7f69ccee0700 1 --2- 172.21.15.88:0/2575512088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69c8106890 0x7f69c81045e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f69bc0118f0 tx=0x7f69bc0119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:49.165 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.625+0000 7f69c5ffb700 1 -- 172.21.15.88:0/2575512088 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69bc0036a0 con 0x7f69c8106890 2023-07-14T17:36:49.165 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.625+0000 7f69c5ffb700 1 -- 172.21.15.88:0/2575512088 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f69bc01b440 con 0x7f69c8106890 2023-07-14T17:36:49.165 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.625+0000 7f69c5ffb700 1 -- 172.21.15.88:0/2575512088 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f69bc0193f0 con 0x7f69c8106890 2023-07-14T17:36:49.165 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.625+0000 7f69cf144700 1 -- 172.21.15.88:0/2575512088 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c8106370 con 0x7f69c8106890 2023-07-14T17:36:49.165 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.625+0000 7f69cf144700 1 -- 172.21.15.88:0/2575512088 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c8104cf0 con 0x7f69c8106890 2023-07-14T17:36:49.165 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.626+0000 7f69c5ffb700 1 -- 172.21.15.88:0/2575512088 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f69bc01b5b0 con 0x7f69c8106890 2023-07-14T17:36:49.166 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.626+0000 7f69c5ffb700 1 --2- 172.21.15.88:0/2575512088 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f69b8038760 0x7f69b803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:49.166 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.626+0000 7f69c5ffb700 1 -- 172.21.15.88:0/2575512088 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f69bc04c770 con 0x7f69c8106890 2023-07-14T17:36:49.166 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.626+0000 7f69cf144700 1 -- 172.21.15.88:0/2575512088 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69c8061c70 con 0x7f69c8106890 2023-07-14T17:36:49.166 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.626+0000 7f69c7fff700 1 --2- 172.21.15.88:0/2575512088 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f69b8038760 0x7f69b803ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:49.166 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.626+0000 7f69c7fff700 1 --2- 172.21.15.88:0/2575512088 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f69b8038760 0x7f69b803ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f69b4006fd0 tx=0x7f69b4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:49.166 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.631+0000 7f69c5ffb700 1 -- 172.21.15.88:0/2575512088 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f69bc01f020 con 0x7f69c8106890 2023-07-14T17:36:49.167 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.817+0000 7f69cf144700 1 -- 172.21.15.88:0/2575512088 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f69c81051e0 con 0x7f69c8106890 2023-07-14T17:36:49.167 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.817+0000 7f69c5ffb700 1 -- 172.21.15.88:0/2575512088 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f69bc023590 con 0x7f69c8106890 2023-07-14T17:36:49.167 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.819+0000 7f69cf144700 1 -- 172.21.15.88:0/2575512088 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f69b8038760 msgr2=0x7f69b803ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:49.167 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.819+0000 7f69cf144700 1 --2- 172.21.15.88:0/2575512088 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f69b8038760 0x7f69b803ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f69b4006fd0 tx=0x7f69b4006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:49.167 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.819+0000 7f69cf144700 1 -- 172.21.15.88:0/2575512088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69c8106890 msgr2=0x7f69c81045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:49.167 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.819+0000 7f69cf144700 1 --2- 172.21.15.88:0/2575512088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69c8106890 0x7f69c81045e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f69bc0118f0 tx=0x7f69bc0119d0 comp rx=0 tx=0).stop 2023-07-14T17:36:49.167 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.819+0000 7f69cf144700 1 -- 172.21.15.88:0/2575512088 shutdown_connections 2023-07-14T17:36:49.167 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.819+0000 7f69cf144700 1 --2- 172.21.15.88:0/2575512088 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f69b8038760 0x7f69b803ac10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:49.168 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.819+0000 7f69cf144700 1 --2- 172.21.15.88:0/2575512088 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f69c8106890 0x7f69c81045e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:49.168 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.819+0000 7f69cf144700 1 -- 172.21.15.88:0/2575512088 >> 172.21.15.88:0/2575512088 conn(0x7f69c80feec0 msgr2=0x7f69c80ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:49.168 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.819+0000 7f69cf144700 1 -- 172.21.15.88:0/2575512088 shutdown_connections 2023-07-14T17:36:49.168 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:48.819+0000 7f69cf144700 1 -- 172.21.15.88:0/2575512088 wait complete. 2023-07-14T17:36:49.169 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T17:36:49.169 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout disabled 2023-07-14T17:36:49.171 INFO:teuthology.orchestra.run.smithi088.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T17:36:49.172 INFO:teuthology.orchestra.run.smithi088.stdout:systemctl: stdout inactive 2023-07-14T17:36:49.172 INFO:teuthology.orchestra.run.smithi088.stdout:firewalld.service is not enabled 2023-07-14T17:36:49.172 INFO:teuthology.orchestra.run.smithi088.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-14T17:36:49.173 INFO:teuthology.orchestra.run.smithi088.stdout:Ceph Dashboard is now available at: 2023-07-14T17:36:49.174 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:36:49.174 INFO:teuthology.orchestra.run.smithi088.stdout: URL: https://smithi088.front.sepia.ceph.com:8443/ 2023-07-14T17:36:49.174 INFO:teuthology.orchestra.run.smithi088.stdout: User: admin 2023-07-14T17:36:49.174 INFO:teuthology.orchestra.run.smithi088.stdout: Password: ptp5qoalrd 2023-07-14T17:36:49.175 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:36:49.175 INFO:teuthology.orchestra.run.smithi088.stdout:Saving cluster configuration to /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config directory 2023-07-14T17:36:49.175 INFO:teuthology.orchestra.run.smithi088.stdout:Enabling autotune for osd_memory_target 2023-07-14T17:36:49.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:49 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2575512088' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-14T17:36:50.217 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.652+0000 7f70ff70f700 1 Processor -- start 2023-07-14T17:36:50.218 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.652+0000 7f70ff70f700 1 -- start start 2023-07-14T17:36:50.218 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.652+0000 7f70ff70f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70f81051f0 0x7f70f8105600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:50.218 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.652+0000 7f70ff70f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f8105b40 con 0x7f70f81051f0 2023-07-14T17:36:50.218 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.653+0000 7f70fd4ab700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70f81051f0 0x7f70f8105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:50.218 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.653+0000 7f70fd4ab700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70f81051f0 0x7f70f8105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40434/0 (socket says 172.21.15.88:40434) 2023-07-14T17:36:50.218 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.653+0000 7f70fd4ab700 1 -- 172.21.15.88:0/3607859389 learned_addr learned my addr 172.21.15.88:0/3607859389 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:50.219 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.653+0000 7f70fd4ab700 1 -- 172.21.15.88:0/3607859389 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f8105c80 con 0x7f70f81051f0 2023-07-14T17:36:50.219 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.653+0000 7f70fd4ab700 1 --2- 172.21.15.88:0/3607859389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70f81051f0 0x7f70f8105600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f70e80098a0 tx=0x7f70e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f747c9413a4bca server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:50.219 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.653+0000 7f70effff700 1 -- 172.21.15.88:0/3607859389 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f70e8004160 con 0x7f70f81051f0 2023-07-14T17:36:50.219 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.653+0000 7f70effff700 1 -- 172.21.15.88:0/3607859389 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f70e80042c0 con 0x7f70f81051f0 2023-07-14T17:36:50.219 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.653+0000 7f70effff700 1 -- 172.21.15.88:0/3607859389 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f70e80118e0 con 0x7f70f81051f0 2023-07-14T17:36:50.219 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.654+0000 7f70ff70f700 1 -- 172.21.15.88:0/3607859389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70f81051f0 msgr2=0x7f70f8105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:50.219 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.654+0000 7f70ff70f700 1 --2- 172.21.15.88:0/3607859389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70f81051f0 0x7f70f8105600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f70e80098a0 tx=0x7f70e8009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:50.220 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.654+0000 7f70ff70f700 1 -- 172.21.15.88:0/3607859389 shutdown_connections 2023-07-14T17:36:50.220 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.654+0000 7f70ff70f700 1 --2- 172.21.15.88:0/3607859389 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70f81051f0 0x7f70f8105600 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:50.220 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.654+0000 7f70ff70f700 1 -- 172.21.15.88:0/3607859389 >> 172.21.15.88:0/3607859389 conn(0x7f70f81008c0 msgr2=0x7f70f8102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:50.220 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.654+0000 7f70ff70f700 1 -- 172.21.15.88:0/3607859389 shutdown_connections 2023-07-14T17:36:50.220 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.654+0000 7f70ff70f700 1 -- 172.21.15.88:0/3607859389 wait complete. 2023-07-14T17:36:50.220 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.654+0000 7f70ff70f700 1 Processor -- start 2023-07-14T17:36:50.220 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.654+0000 7f70ff70f700 1 -- start start 2023-07-14T17:36:50.221 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.655+0000 7f70ff70f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70f81051f0 0x7f70f807a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:50.221 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.655+0000 7f70ff70f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f807be00 con 0x7f70f81051f0 2023-07-14T17:36:50.221 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.655+0000 7f70fd4ab700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70f81051f0 0x7f70f807a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:50.221 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.655+0000 7f70fd4ab700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70f81051f0 0x7f70f807a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40436/0 (socket says 172.21.15.88:40436) 2023-07-14T17:36:50.221 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.655+0000 7f70fd4ab700 1 -- 172.21.15.88:0/1558301936 learned_addr learned my addr 172.21.15.88:0/1558301936 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:50.221 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.655+0000 7f70fd4ab700 1 -- 172.21.15.88:0/1558301936 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e8009550 con 0x7f70f81051f0 2023-07-14T17:36:50.221 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.655+0000 7f70fd4ab700 1 --2- 172.21.15.88:0/1558301936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70f81051f0 0x7f70f807a270 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f70e8003e30 tx=0x7f70e80119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:50.222 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.655+0000 7f70ee7fc700 1 -- 172.21.15.88:0/1558301936 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f70e80036a0 con 0x7f70f81051f0 2023-07-14T17:36:50.222 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.655+0000 7f70ee7fc700 1 -- 172.21.15.88:0/1558301936 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f70e801b440 con 0x7f70f81051f0 2023-07-14T17:36:50.222 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.655+0000 7f70ff70f700 1 -- 172.21.15.88:0/1558301936 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f807c000 con 0x7f70f81051f0 2023-07-14T17:36:50.222 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.655+0000 7f70ee7fc700 1 -- 172.21.15.88:0/1558301936 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f70e8023400 con 0x7f70f81051f0 2023-07-14T17:36:50.222 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.655+0000 7f70ff70f700 1 -- 172.21.15.88:0/1558301936 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f807a980 con 0x7f70f81051f0 2023-07-14T17:36:50.222 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.656+0000 7f70ee7fc700 1 -- 172.21.15.88:0/1558301936 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f70e8003800 con 0x7f70f81051f0 2023-07-14T17:36:50.222 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.656+0000 7f70ee7fc700 1 --2- 172.21.15.88:0/1558301936 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f70e40387b0 0x7f70e403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:50.222 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.656+0000 7f70ff70f700 1 -- 172.21.15.88:0/1558301936 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f8061c70 con 0x7f70f81051f0 2023-07-14T17:36:50.223 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.656+0000 7f70ee7fc700 1 -- 172.21.15.88:0/1558301936 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f70e804c760 con 0x7f70f81051f0 2023-07-14T17:36:50.223 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.658+0000 7f70fccaa700 1 --2- 172.21.15.88:0/1558301936 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f70e40387b0 0x7f70e403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:50.223 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.658+0000 7f70fccaa700 1 --2- 172.21.15.88:0/1558301936 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f70e40387b0 0x7f70e403ac60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f70f4006fd0 tx=0x7f70f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:50.223 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.661+0000 7f70ee7fc700 1 -- 172.21.15.88:0/1558301936 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f70e802b740 con 0x7f70f81051f0 2023-07-14T17:36:50.223 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.846+0000 7f70ff70f700 1 -- 172.21.15.88:0/1558301936 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f70f807b120 con 0x7f70f81051f0 2023-07-14T17:36:50.223 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.847+0000 7f70ee7fc700 1 -- 172.21.15.88:0/1558301936 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f70e802b350 con 0x7f70f81051f0 2023-07-14T17:36:50.223 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.849+0000 7f70ff70f700 1 -- 172.21.15.88:0/1558301936 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f70e40387b0 msgr2=0x7f70e403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:50.223 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.849+0000 7f70ff70f700 1 --2- 172.21.15.88:0/1558301936 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f70e40387b0 0x7f70e403ac60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f70f4006fd0 tx=0x7f70f4006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:50.224 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.849+0000 7f70ff70f700 1 -- 172.21.15.88:0/1558301936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70f81051f0 msgr2=0x7f70f807a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:50.224 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.849+0000 7f70ff70f700 1 --2- 172.21.15.88:0/1558301936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70f81051f0 0x7f70f807a270 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f70e8003e30 tx=0x7f70e80119a0 comp rx=0 tx=0).stop 2023-07-14T17:36:50.224 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.849+0000 7f70ff70f700 1 -- 172.21.15.88:0/1558301936 shutdown_connections 2023-07-14T17:36:50.224 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.849+0000 7f70ff70f700 1 --2- 172.21.15.88:0/1558301936 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f70e40387b0 0x7f70e403ac60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:50.224 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.849+0000 7f70ff70f700 1 --2- 172.21.15.88:0/1558301936 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70f81051f0 0x7f70f807a270 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:50.224 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.849+0000 7f70ff70f700 1 -- 172.21.15.88:0/1558301936 >> 172.21.15.88:0/1558301936 conn(0x7f70f81008c0 msgr2=0x7f70f8101f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:50.224 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.849+0000 7f70ff70f700 1 -- 172.21.15.88:0/1558301936 shutdown_connections 2023-07-14T17:36:50.225 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:49.849+0000 7f70ff70f700 1 -- 172.21.15.88:0/1558301936 wait complete. 2023-07-14T17:36:51.464 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.734+0000 7ff0ba922700 1 Processor -- start 2023-07-14T17:36:51.464 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.734+0000 7ff0ba922700 1 -- start start 2023-07-14T17:36:51.464 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.734+0000 7ff0ba922700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0b41040e0 0x7ff0b41044f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:51.464 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.734+0000 7ff0ba922700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b4104a30 con 0x7ff0b41040e0 2023-07-14T17:36:51.464 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.734+0000 7ff0b3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0b41040e0 0x7ff0b41044f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:51.465 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.734+0000 7ff0b3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0b41040e0 0x7ff0b41044f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40440/0 (socket says 172.21.15.88:40440) 2023-07-14T17:36:51.465 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.734+0000 7ff0b3fff700 1 -- 172.21.15.88:0/832703171 learned_addr learned my addr 172.21.15.88:0/832703171 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:51.465 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.735+0000 7ff0b3fff700 1 -- 172.21.15.88:0/832703171 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b4104b70 con 0x7ff0b41040e0 2023-07-14T17:36:51.465 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.735+0000 7ff0b3fff700 1 --2- 172.21.15.88:0/832703171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0b41040e0 0x7ff0b41044f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0098a0 tx=0x7ff09c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8797bb51d6b930af server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:51.465 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.735+0000 7ff0b2ffd700 1 -- 172.21.15.88:0/832703171 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff09c004160 con 0x7ff0b41040e0 2023-07-14T17:36:51.465 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.735+0000 7ff0b2ffd700 1 -- 172.21.15.88:0/832703171 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff09c0042c0 con 0x7ff0b41040e0 2023-07-14T17:36:51.465 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.735+0000 7ff0b2ffd700 1 -- 172.21.15.88:0/832703171 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff09c0118e0 con 0x7ff0b41040e0 2023-07-14T17:36:51.466 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.735+0000 7ff0ba922700 1 -- 172.21.15.88:0/832703171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0b41040e0 msgr2=0x7ff0b41044f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:51.466 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.735+0000 7ff0ba922700 1 --2- 172.21.15.88:0/832703171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0b41040e0 0x7ff0b41044f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0098a0 tx=0x7ff09c009bb0 comp rx=0 tx=0).stop 2023-07-14T17:36:51.466 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.736+0000 7ff0ba922700 1 -- 172.21.15.88:0/832703171 shutdown_connections 2023-07-14T17:36:51.466 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.736+0000 7ff0ba922700 1 --2- 172.21.15.88:0/832703171 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0b41040e0 0x7ff0b41044f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:51.466 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.736+0000 7ff0ba922700 1 -- 172.21.15.88:0/832703171 >> 172.21.15.88:0/832703171 conn(0x7ff0b40ff750 msgr2=0x7ff0b4101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:51.466 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.736+0000 7ff0ba922700 1 -- 172.21.15.88:0/832703171 shutdown_connections 2023-07-14T17:36:51.466 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.736+0000 7ff0ba922700 1 -- 172.21.15.88:0/832703171 wait complete. 2023-07-14T17:36:51.467 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.736+0000 7ff0ba922700 1 Processor -- start 2023-07-14T17:36:51.467 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.736+0000 7ff0ba922700 1 -- start start 2023-07-14T17:36:51.467 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.736+0000 7ff0ba922700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0b41040e0 0x7ff0b4197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:51.467 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.736+0000 7ff0ba922700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b41983d0 con 0x7ff0b41040e0 2023-07-14T17:36:51.467 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.737+0000 7ff0b3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0b41040e0 0x7ff0b4197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:51.467 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.737+0000 7ff0b3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0b41040e0 0x7ff0b4197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40442/0 (socket says 172.21.15.88:40442) 2023-07-14T17:36:51.467 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.737+0000 7ff0b3fff700 1 -- 172.21.15.88:0/1158098904 learned_addr learned my addr 172.21.15.88:0/1158098904 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:51.467 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.737+0000 7ff0b3fff700 1 -- 172.21.15.88:0/1158098904 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09c009550 con 0x7ff0b41040e0 2023-07-14T17:36:51.468 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.737+0000 7ff0b3fff700 1 --2- 172.21.15.88:0/1158098904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0b41040e0 0x7ff0b4197e90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0118f0 tx=0x7ff09c0119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:51.468 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.737+0000 7ff0b17fa700 1 -- 172.21.15.88:0/1158098904 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff09c0036a0 con 0x7ff0b41040e0 2023-07-14T17:36:51.468 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.737+0000 7ff0b17fa700 1 -- 172.21.15.88:0/1158098904 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff09c01b440 con 0x7ff0b41040e0 2023-07-14T17:36:51.468 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.737+0000 7ff0b17fa700 1 -- 172.21.15.88:0/1158098904 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff09c0193f0 con 0x7ff0b41040e0 2023-07-14T17:36:51.468 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.737+0000 7ff0ba922700 1 -- 172.21.15.88:0/1158098904 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b41985d0 con 0x7ff0b41040e0 2023-07-14T17:36:51.468 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.737+0000 7ff0ba922700 1 -- 172.21.15.88:0/1158098904 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b41989f0 con 0x7ff0b41040e0 2023-07-14T17:36:51.468 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.738+0000 7ff0b17fa700 1 -- 172.21.15.88:0/1158098904 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7ff09c019660 con 0x7ff0b41040e0 2023-07-14T17:36:51.469 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.738+0000 7ff0b17fa700 1 --2- 172.21.15.88:0/1158098904 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff0a0038850 0x7ff0a003ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:51.469 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.738+0000 7ff0b17fa700 1 -- 172.21.15.88:0/1158098904 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff09c04ca10 con 0x7ff0b41040e0 2023-07-14T17:36:51.469 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.738+0000 7ff0ba922700 1 -- 172.21.15.88:0/1158098904 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b4068520 con 0x7ff0b41040e0 2023-07-14T17:36:51.469 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.738+0000 7ff0b37fe700 1 --2- 172.21.15.88:0/1158098904 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff0a0038850 0x7ff0a003ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:51.469 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.739+0000 7ff0b37fe700 1 --2- 172.21.15.88:0/1158098904 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff0a0038850 0x7ff0a003ad00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4006fd0 tx=0x7ff0a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:51.469 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:50.743+0000 7ff0b17fa700 1 -- 172.21.15.88:0/1158098904 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff09c0239a0 con 0x7ff0b41040e0 2023-07-14T17:36:51.469 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:51.032+0000 7ff0ba922700 1 -- 172.21.15.88:0/1158098904 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7ff0b4105f80 con 0x7ff0b41040e0 2023-07-14T17:36:51.470 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:51.037+0000 7ff0b17fa700 1 -- 172.21.15.88:0/1158098904 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7ff09c0233b0 con 0x7ff0b41040e0 2023-07-14T17:36:51.470 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:51.039+0000 7ff0ba922700 1 -- 172.21.15.88:0/1158098904 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff0a0038850 msgr2=0x7ff0a003ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:51.470 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:51.039+0000 7ff0ba922700 1 --2- 172.21.15.88:0/1158098904 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff0a0038850 0x7ff0a003ad00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4006fd0 tx=0x7ff0a4006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:51.470 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:51.039+0000 7ff0ba922700 1 -- 172.21.15.88:0/1158098904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0b41040e0 msgr2=0x7ff0b4197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:51.470 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:51.039+0000 7ff0ba922700 1 --2- 172.21.15.88:0/1158098904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0b41040e0 0x7ff0b4197e90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff09c0118f0 tx=0x7ff09c0119d0 comp rx=0 tx=0).stop 2023-07-14T17:36:51.470 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:51.039+0000 7ff0ba922700 1 -- 172.21.15.88:0/1158098904 shutdown_connections 2023-07-14T17:36:51.470 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:51.039+0000 7ff0ba922700 1 --2- 172.21.15.88:0/1158098904 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff0a0038850 0x7ff0a003ad00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:51.470 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:51.039+0000 7ff0ba922700 1 --2- 172.21.15.88:0/1158098904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0b41040e0 0x7ff0b4197e90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:51.471 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:51.039+0000 7ff0ba922700 1 -- 172.21.15.88:0/1158098904 >> 172.21.15.88:0/1158098904 conn(0x7ff0b40ff750 msgr2=0x7ff0b4100500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:51.471 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:51.039+0000 7ff0ba922700 1 -- 172.21.15.88:0/1158098904 shutdown_connections 2023-07-14T17:36:51.471 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr 2023-07-14T17:36:51.039+0000 7ff0ba922700 1 -- 172.21.15.88:0/1158098904 wait complete. 2023-07-14T17:36:51.471 INFO:teuthology.orchestra.run.smithi088.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-07-14T17:36:51.471 INFO:teuthology.orchestra.run.smithi088.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-07-14T17:36:51.471 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:36:51.471 INFO:teuthology.orchestra.run.smithi088.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-14T17:36:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:36:51.472 INFO:teuthology.orchestra.run.smithi088.stdout:Or, if you are only running a single cluster on this host: 2023-07-14T17:36:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:36:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-07-14T17:36:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:36:51.472 INFO:teuthology.orchestra.run.smithi088.stdout:Please consider enabling telemetry to help improve Ceph: 2023-07-14T17:36:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:36:51.472 INFO:teuthology.orchestra.run.smithi088.stdout: ceph telemetry on 2023-07-14T17:36:51.473 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:36:51.473 INFO:teuthology.orchestra.run.smithi088.stdout:For more information see: 2023-07-14T17:36:51.473 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:36:51.473 INFO:teuthology.orchestra.run.smithi088.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-07-14T17:36:51.473 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:36:51.473 INFO:teuthology.orchestra.run.smithi088.stdout:Bootstrap complete. 2023-07-14T17:36:51.500 INFO:tasks.cephadm:Fetching config... 2023-07-14T17:36:51.500 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:36:51.501 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-14T17:36:51.517 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-14T17:36:51.518 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:36:51.518 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-14T17:36:51.574 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-14T17:36:51.574 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:36:51.575 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-14T17:36:51.643 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-14T17:36:51.644 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:36:51.644 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-14T17:36:51.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:51 smithi088 ceph-mon[127465]: mgrmap e13: x(active, since 6s) 2023-07-14T17:36:51.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:51 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1158098904' entity='client.admin' 2023-07-14T17:36:51.699 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-14T17:36:51.700 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM2T4TXzNTgec4y9vvL8s6pT+SV6AoSJaL6ILEF/+YMDh2EGtE+GNYPLjpg00uX/xx3g5HM9j3rDYngdrGR3Ao8o9BhjVszmZbfCeyKg5ysOmdKKnuSclx3crkfXVMcrNZaHTdlhoF/EaoBwQJYNfn7amGcDiavcE8MX/BLHjw97MUtRVu+nyEesI4AFu0V4vG31XwdCWXmeG2qBbl6KuXZAntns/w5EbhmAaXWPk9gvs8rbeSUp9FZ4SjzZ0cfY2Ho6lWmCVdo0gIOvYi0RUX1AdQ8qisQ71ommeeR7iFg3IhuChlgTIAN7kTDvmajR5cF71+mk8rpOopdc212G6QlYbFgPrUTcUmFYxCEssvmDiplgd5rZ59m15q4mOxbEEN8dkdvpJ0sL57jHylCkuSkrrAJ+FoVOEcj/pyq0tOya86dYwE4TUrYfmblyn6dlllvcVD23Q49E7BSuHr9DTeQlot6iPGg4A7UPPg7JLu7zw7v3m/A2D+MrGQP8szRC8= ceph-b462a556-226c-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T17:36:51.775 INFO:teuthology.orchestra.run.smithi088.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM2T4TXzNTgec4y9vvL8s6pT+SV6AoSJaL6ILEF/+YMDh2EGtE+GNYPLjpg00uX/xx3g5HM9j3rDYngdrGR3Ao8o9BhjVszmZbfCeyKg5ysOmdKKnuSclx3crkfXVMcrNZaHTdlhoF/EaoBwQJYNfn7amGcDiavcE8MX/BLHjw97MUtRVu+nyEesI4AFu0V4vG31XwdCWXmeG2qBbl6KuXZAntns/w5EbhmAaXWPk9gvs8rbeSUp9FZ4SjzZ0cfY2Ho6lWmCVdo0gIOvYi0RUX1AdQ8qisQ71ommeeR7iFg3IhuChlgTIAN7kTDvmajR5cF71+mk8rpOopdc212G6QlYbFgPrUTcUmFYxCEssvmDiplgd5rZ59m15q4mOxbEEN8dkdvpJ0sL57jHylCkuSkrrAJ+FoVOEcj/pyq0tOya86dYwE4TUrYfmblyn6dlllvcVD23Q49E7BSuHr9DTeQlot6iPGg4A7UPPg7JLu7zw7v3m/A2D+MrGQP8szRC8= ceph-b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:36:51.789 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM2T4TXzNTgec4y9vvL8s6pT+SV6AoSJaL6ILEF/+YMDh2EGtE+GNYPLjpg00uX/xx3g5HM9j3rDYngdrGR3Ao8o9BhjVszmZbfCeyKg5ysOmdKKnuSclx3crkfXVMcrNZaHTdlhoF/EaoBwQJYNfn7amGcDiavcE8MX/BLHjw97MUtRVu+nyEesI4AFu0V4vG31XwdCWXmeG2qBbl6KuXZAntns/w5EbhmAaXWPk9gvs8rbeSUp9FZ4SjzZ0cfY2Ho6lWmCVdo0gIOvYi0RUX1AdQ8qisQ71ommeeR7iFg3IhuChlgTIAN7kTDvmajR5cF71+mk8rpOopdc212G6QlYbFgPrUTcUmFYxCEssvmDiplgd5rZ59m15q4mOxbEEN8dkdvpJ0sL57jHylCkuSkrrAJ+FoVOEcj/pyq0tOya86dYwE4TUrYfmblyn6dlllvcVD23Q49E7BSuHr9DTeQlot6iPGg4A7UPPg7JLu7zw7v3m/A2D+MrGQP8szRC8= ceph-b462a556-226c-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T17:36:51.834 INFO:teuthology.orchestra.run.smithi107.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM2T4TXzNTgec4y9vvL8s6pT+SV6AoSJaL6ILEF/+YMDh2EGtE+GNYPLjpg00uX/xx3g5HM9j3rDYngdrGR3Ao8o9BhjVszmZbfCeyKg5ysOmdKKnuSclx3crkfXVMcrNZaHTdlhoF/EaoBwQJYNfn7amGcDiavcE8MX/BLHjw97MUtRVu+nyEesI4AFu0V4vG31XwdCWXmeG2qBbl6KuXZAntns/w5EbhmAaXWPk9gvs8rbeSUp9FZ4SjzZ0cfY2Ho6lWmCVdo0gIOvYi0RUX1AdQ8qisQ71ommeeR7iFg3IhuChlgTIAN7kTDvmajR5cF71+mk8rpOopdc212G6QlYbFgPrUTcUmFYxCEssvmDiplgd5rZ59m15q4mOxbEEN8dkdvpJ0sL57jHylCkuSkrrAJ+FoVOEcj/pyq0tOya86dYwE4TUrYfmblyn6dlllvcVD23Q49E7BSuHr9DTeQlot6iPGg4A7UPPg7JLu7zw7v3m/A2D+MrGQP8szRC8= ceph-b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:36:51.846 DEBUG:teuthology.orchestra.run.smithi178:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM2T4TXzNTgec4y9vvL8s6pT+SV6AoSJaL6ILEF/+YMDh2EGtE+GNYPLjpg00uX/xx3g5HM9j3rDYngdrGR3Ao8o9BhjVszmZbfCeyKg5ysOmdKKnuSclx3crkfXVMcrNZaHTdlhoF/EaoBwQJYNfn7amGcDiavcE8MX/BLHjw97MUtRVu+nyEesI4AFu0V4vG31XwdCWXmeG2qBbl6KuXZAntns/w5EbhmAaXWPk9gvs8rbeSUp9FZ4SjzZ0cfY2Ho6lWmCVdo0gIOvYi0RUX1AdQ8qisQ71ommeeR7iFg3IhuChlgTIAN7kTDvmajR5cF71+mk8rpOopdc212G6QlYbFgPrUTcUmFYxCEssvmDiplgd5rZ59m15q4mOxbEEN8dkdvpJ0sL57jHylCkuSkrrAJ+FoVOEcj/pyq0tOya86dYwE4TUrYfmblyn6dlllvcVD23Q49E7BSuHr9DTeQlot6iPGg4A7UPPg7JLu7zw7v3m/A2D+MrGQP8szRC8= ceph-b462a556-226c-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T17:36:51.891 INFO:teuthology.orchestra.run.smithi178.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM2T4TXzNTgec4y9vvL8s6pT+SV6AoSJaL6ILEF/+YMDh2EGtE+GNYPLjpg00uX/xx3g5HM9j3rDYngdrGR3Ao8o9BhjVszmZbfCeyKg5ysOmdKKnuSclx3crkfXVMcrNZaHTdlhoF/EaoBwQJYNfn7amGcDiavcE8MX/BLHjw97MUtRVu+nyEesI4AFu0V4vG31XwdCWXmeG2qBbl6KuXZAntns/w5EbhmAaXWPk9gvs8rbeSUp9FZ4SjzZ0cfY2Ho6lWmCVdo0gIOvYi0RUX1AdQ8qisQ71ommeeR7iFg3IhuChlgTIAN7kTDvmajR5cF71+mk8rpOopdc212G6QlYbFgPrUTcUmFYxCEssvmDiplgd5rZ59m15q4mOxbEEN8dkdvpJ0sL57jHylCkuSkrrAJ+FoVOEcj/pyq0tOya86dYwE4TUrYfmblyn6dlllvcVD23Q49E7BSuHr9DTeQlot6iPGg4A7UPPg7JLu7zw7v3m/A2D+MrGQP8szRC8= ceph-b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:36:51.904 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-07-14T17:36:52.463 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:36:54.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.253+0000 7f1e62a90700 1 -- 172.21.15.88:0/2365816011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e5c0ff720 msgr2=0x7f1e5c0ffb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:54.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.253+0000 7f1e62a90700 1 --2- 172.21.15.88:0/2365816011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e5c0ff720 0x7f1e5c0ffb30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1e50009910 tx=0x7f1e50009c20 comp rx=0 tx=0).stop 2023-07-14T17:36:54.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.253+0000 7f1e62a90700 1 -- 172.21.15.88:0/2365816011 shutdown_connections 2023-07-14T17:36:54.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.253+0000 7f1e62a90700 1 --2- 172.21.15.88:0/2365816011 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e5c0ff720 0x7f1e5c0ffb30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:54.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.253+0000 7f1e62a90700 1 -- 172.21.15.88:0/2365816011 >> 172.21.15.88:0/2365816011 conn(0x7f1e5c0facb0 msgr2=0x7f1e5c0fd100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:54.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.254+0000 7f1e62a90700 1 -- 172.21.15.88:0/2365816011 shutdown_connections 2023-07-14T17:36:54.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.254+0000 7f1e62a90700 1 -- 172.21.15.88:0/2365816011 wait complete. 2023-07-14T17:36:54.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.254+0000 7f1e62a90700 1 Processor -- start 2023-07-14T17:36:54.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.254+0000 7f1e62a90700 1 -- start start 2023-07-14T17:36:54.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.255+0000 7f1e62a90700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e5c0ff720 0x7f1e5c1947a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:54.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.255+0000 7f1e62a90700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5c194ce0 con 0x7f1e5c0ff720 2023-07-14T17:36:54.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.255+0000 7f1e6082c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e5c0ff720 0x7f1e5c1947a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:54.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.255+0000 7f1e6082c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e5c0ff720 0x7f1e5c1947a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40448/0 (socket says 172.21.15.88:40448) 2023-07-14T17:36:54.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.255+0000 7f1e6082c700 1 -- 172.21.15.88:0/1481269917 learned_addr learned my addr 172.21.15.88:0/1481269917 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:54.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.255+0000 7f1e6082c700 1 -- 172.21.15.88:0/1481269917 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e500095f0 con 0x7f1e5c0ff720 2023-07-14T17:36:54.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.255+0000 7f1e6082c700 1 --2- 172.21.15.88:0/1481269917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e5c0ff720 0x7f1e5c1947a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1e50006010 tx=0x7f1e50005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:54.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.255+0000 7f1e59ffb700 1 -- 172.21.15.88:0/1481269917 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1e5000ba90 con 0x7f1e5c0ff720 2023-07-14T17:36:54.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.255+0000 7f1e59ffb700 1 -- 172.21.15.88:0/1481269917 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1e5000bbf0 con 0x7f1e5c0ff720 2023-07-14T17:36:54.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.255+0000 7f1e62a90700 1 -- 172.21.15.88:0/1481269917 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e5c194ee0 con 0x7f1e5c0ff720 2023-07-14T17:36:54.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.255+0000 7f1e59ffb700 1 -- 172.21.15.88:0/1481269917 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1e50017440 con 0x7f1e5c0ff720 2023-07-14T17:36:54.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.255+0000 7f1e62a90700 1 -- 172.21.15.88:0/1481269917 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e5c195380 con 0x7f1e5c0ff720 2023-07-14T17:36:54.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.256+0000 7f1e59ffb700 1 -- 172.21.15.88:0/1481269917 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f1e50017670 con 0x7f1e5c0ff720 2023-07-14T17:36:54.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.256+0000 7f1e62a90700 1 -- 172.21.15.88:0/1481269917 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e5c18e1a0 con 0x7f1e5c0ff720 2023-07-14T17:36:54.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.256+0000 7f1e59ffb700 1 --2- 172.21.15.88:0/1481269917 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1e4c0387b0 0x7f1e4c03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:54.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.257+0000 7f1e59ffb700 1 -- 172.21.15.88:0/1481269917 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1e5004c980 con 0x7f1e5c0ff720 2023-07-14T17:36:54.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.257+0000 7f1e5bfff700 1 --2- 172.21.15.88:0/1481269917 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1e4c0387b0 0x7f1e4c03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:54.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.257+0000 7f1e5bfff700 1 --2- 172.21.15.88:0/1481269917 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1e4c0387b0 0x7f1e4c03ac60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1e48006fd0 tx=0x7f1e48006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:54.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.261+0000 7f1e59ffb700 1 -- 172.21.15.88:0/1481269917 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1e5002ab50 con 0x7f1e5c0ff720 2023-07-14T17:36:54.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.446+0000 7f1e62a90700 1 -- 172.21.15.88:0/1481269917 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f1e5c02c8e0 con 0x7f1e5c0ff720 2023-07-14T17:36:54.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.449+0000 7f1e59ffb700 1 -- 172.21.15.88:0/1481269917 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f1e50027030 con 0x7f1e5c0ff720 2023-07-14T17:36:54.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.451+0000 7f1e62a90700 1 -- 172.21.15.88:0/1481269917 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1e4c0387b0 msgr2=0x7f1e4c03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:54.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.451+0000 7f1e62a90700 1 --2- 172.21.15.88:0/1481269917 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1e4c0387b0 0x7f1e4c03ac60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1e48006fd0 tx=0x7f1e48006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:54.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.451+0000 7f1e62a90700 1 -- 172.21.15.88:0/1481269917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e5c0ff720 msgr2=0x7f1e5c1947a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:54.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.451+0000 7f1e62a90700 1 --2- 172.21.15.88:0/1481269917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e5c0ff720 0x7f1e5c1947a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1e50006010 tx=0x7f1e50005810 comp rx=0 tx=0).stop 2023-07-14T17:36:54.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.452+0000 7f1e62a90700 1 -- 172.21.15.88:0/1481269917 shutdown_connections 2023-07-14T17:36:54.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.452+0000 7f1e62a90700 1 --2- 172.21.15.88:0/1481269917 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1e4c0387b0 0x7f1e4c03ac60 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:54.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.452+0000 7f1e62a90700 1 --2- 172.21.15.88:0/1481269917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e5c0ff720 0x7f1e5c1947a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:54.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.452+0000 7f1e62a90700 1 -- 172.21.15.88:0/1481269917 >> 172.21.15.88:0/1481269917 conn(0x7f1e5c0facb0 msgr2=0x7f1e5c0fba40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:54.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.452+0000 7f1e62a90700 1 -- 172.21.15.88:0/1481269917 shutdown_connections 2023-07-14T17:36:54.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:54.452+0000 7f1e62a90700 1 -- 172.21.15.88:0/1481269917 wait complete. 2023-07-14T17:36:54.827 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-07-14T17:36:54.827 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-07-14T17:36:55.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:55 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:36:55.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:55 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:36:55.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:55 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi088", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:36:55.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:55 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:36:55.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:55 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:36:55.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:55 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:36:55.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:55 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1481269917' entity='client.admin' 2023-07-14T17:36:55.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:55 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:36:55.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:55 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:36:55.381 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:36:56.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.580+0000 7fbeafe9b700 1 -- 172.21.15.88:0/883803334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea80fe610 msgr2=0x7fbea80fea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:56.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.580+0000 7fbeafe9b700 1 --2- 172.21.15.88:0/883803334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea80fe610 0x7fbea80fea20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbe98009910 tx=0x7fbe98009c20 comp rx=0 tx=0).stop 2023-07-14T17:36:56.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.580+0000 7fbeafe9b700 1 -- 172.21.15.88:0/883803334 shutdown_connections 2023-07-14T17:36:56.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.580+0000 7fbeafe9b700 1 --2- 172.21.15.88:0/883803334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea80fe610 0x7fbea80fea20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:56.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.580+0000 7fbeafe9b700 1 -- 172.21.15.88:0/883803334 >> 172.21.15.88:0/883803334 conn(0x7fbea80f9ba0 msgr2=0x7fbea80fbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:56.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.580+0000 7fbeafe9b700 1 -- 172.21.15.88:0/883803334 shutdown_connections 2023-07-14T17:36:56.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.580+0000 7fbeafe9b700 1 -- 172.21.15.88:0/883803334 wait complete. 2023-07-14T17:36:56.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.580+0000 7fbeafe9b700 1 Processor -- start 2023-07-14T17:36:56.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.581+0000 7fbeafe9b700 1 -- start start 2023-07-14T17:36:56.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.581+0000 7fbeafe9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea80fe610 0x7fbea81936c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:56.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.581+0000 7fbeafe9b700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea8193c00 con 0x7fbea80fe610 2023-07-14T17:36:56.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.581+0000 7fbeadc37700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea80fe610 0x7fbea81936c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:56.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.581+0000 7fbeadc37700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea80fe610 0x7fbea81936c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40454/0 (socket says 172.21.15.88:40454) 2023-07-14T17:36:56.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.581+0000 7fbeadc37700 1 -- 172.21.15.88:0/2602972159 learned_addr learned my addr 172.21.15.88:0/2602972159 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:56.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.581+0000 7fbeadc37700 1 -- 172.21.15.88:0/2602972159 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe980095f0 con 0x7fbea80fe610 2023-07-14T17:36:56.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.581+0000 7fbeadc37700 1 --2- 172.21.15.88:0/2602972159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea80fe610 0x7fbea81936c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbe98006010 tx=0x7fbe98005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:56.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.581+0000 7fbe9effd700 1 -- 172.21.15.88:0/2602972159 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe9800ba90 con 0x7fbea80fe610 2023-07-14T17:36:56.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.581+0000 7fbe9effd700 1 -- 172.21.15.88:0/2602972159 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fbe9800bbf0 con 0x7fbea80fe610 2023-07-14T17:36:56.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.581+0000 7fbe9effd700 1 -- 172.21.15.88:0/2602972159 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe98017440 con 0x7fbea80fe610 2023-07-14T17:36:56.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.581+0000 7fbeafe9b700 1 -- 172.21.15.88:0/2602972159 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea8193e00 con 0x7fbea80fe610 2023-07-14T17:36:56.572 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.581+0000 7fbeafe9b700 1 -- 172.21.15.88:0/2602972159 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea81942a0 con 0x7fbea80fe610 2023-07-14T17:36:56.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.582+0000 7fbe9effd700 1 -- 172.21.15.88:0/2602972159 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fbe9800bd60 con 0x7fbea80fe610 2023-07-14T17:36:56.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.582+0000 7fbeafe9b700 1 -- 172.21.15.88:0/2602972159 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea818d090 con 0x7fbea80fe610 2023-07-14T17:36:56.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.583+0000 7fbe9effd700 1 --2- 172.21.15.88:0/2602972159 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbe940387b0 0x7fbe9403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:56.573 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.583+0000 7fbe9effd700 1 -- 172.21.15.88:0/2602972159 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbe9804c970 con 0x7fbea80fe610 2023-07-14T17:36:56.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.583+0000 7fbead436700 1 --2- 172.21.15.88:0/2602972159 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbe940387b0 0x7fbe9403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:56.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.583+0000 7fbead436700 1 --2- 172.21.15.88:0/2602972159 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbe940387b0 0x7fbe9403ac60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbea4006fd0 tx=0x7fbea4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:56.574 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.588+0000 7fbe9effd700 1 -- 172.21.15.88:0/2602972159 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbe9802a7d0 con 0x7fbea80fe610 2023-07-14T17:36:56.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.778+0000 7fbeafe9b700 1 -- 172.21.15.88:0/2602972159 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fbea8060b10 con 0x7fbe940387b0 2023-07-14T17:36:56.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.783+0000 7fbe9effd700 1 -- 172.21.15.88:0/2602972159 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbea8060b10 con 0x7fbe940387b0 2023-07-14T17:36:56.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.784+0000 7fbeafe9b700 1 -- 172.21.15.88:0/2602972159 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbe940387b0 msgr2=0x7fbe9403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:56.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.784+0000 7fbeafe9b700 1 --2- 172.21.15.88:0/2602972159 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbe940387b0 0x7fbe9403ac60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbea4006fd0 tx=0x7fbea4006e40 comp rx=0 tx=0).stop 2023-07-14T17:36:56.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.784+0000 7fbeafe9b700 1 -- 172.21.15.88:0/2602972159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea80fe610 msgr2=0x7fbea81936c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:56.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.784+0000 7fbeafe9b700 1 --2- 172.21.15.88:0/2602972159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea80fe610 0x7fbea81936c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbe98006010 tx=0x7fbe98005810 comp rx=0 tx=0).stop 2023-07-14T17:36:56.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.784+0000 7fbeafe9b700 1 -- 172.21.15.88:0/2602972159 shutdown_connections 2023-07-14T17:36:56.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.784+0000 7fbeafe9b700 1 --2- 172.21.15.88:0/2602972159 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbe940387b0 0x7fbe9403ac60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:56.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.784+0000 7fbeafe9b700 1 --2- 172.21.15.88:0/2602972159 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbea80fe610 0x7fbea81936c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:56.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.784+0000 7fbeafe9b700 1 -- 172.21.15.88:0/2602972159 >> 172.21.15.88:0/2602972159 conn(0x7fbea80f9ba0 msgr2=0x7fbea80fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:56.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.784+0000 7fbeafe9b700 1 -- 172.21.15.88:0/2602972159 shutdown_connections 2023-07-14T17:36:56.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:56.784+0000 7fbeafe9b700 1 -- 172.21.15.88:0/2602972159 wait complete. 2023-07-14T17:36:57.236 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi107 2023-07-14T17:36:57.236 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-07-14T17:36:57.237 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/etc/ceph/ceph.conf 2023-07-14T17:36:57.257 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-07-14T17:36:57.257 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T17:36:57.316 INFO:tasks.cephadm:Adding host smithi107 to orchestrator... 2023-07-14T17:36:57.316 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch host add smithi107 2023-07-14T17:36:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:36:58.131 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:57 smithi088 ceph-mon[127465]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:36:58.131 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:57 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:36:58.132 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:57 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:36:58.132 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:57 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:36:58.132 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:57 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:36:58.132 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:57 smithi088 ceph-mon[127465]: Updating smithi088:/etc/ceph/ceph.conf 2023-07-14T17:36:58.132 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:57 smithi088 ceph-mon[127465]: Updating smithi088:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T17:36:58.132 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:57 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:36:58.133 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:57 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:36:58.133 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:57 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:36:59.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.107+0000 7f1ae1d41700 1 -- 172.21.15.88:0/3729385840 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc104660 msgr2=0x7f1adc104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:36:59.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.107+0000 7f1ae1d41700 1 --2- 172.21.15.88:0/3729385840 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc104660 0x7f1adc104a30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4009910 tx=0x7f1ac4009c20 comp rx=0 tx=0).stop 2023-07-14T17:36:59.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.107+0000 7f1ae1d41700 1 -- 172.21.15.88:0/3729385840 shutdown_connections 2023-07-14T17:36:59.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.107+0000 7f1ae1d41700 1 --2- 172.21.15.88:0/3729385840 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc104660 0x7f1adc104a30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:36:59.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.107+0000 7f1ae1d41700 1 -- 172.21.15.88:0/3729385840 >> 172.21.15.88:0/3729385840 conn(0x7f1adc0743e0 msgr2=0x7f1adc0747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:36:59.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.107+0000 7f1ae1d41700 1 -- 172.21.15.88:0/3729385840 shutdown_connections 2023-07-14T17:36:59.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.107+0000 7f1ae1d41700 1 -- 172.21.15.88:0/3729385840 wait complete. 2023-07-14T17:36:59.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.108+0000 7f1ae1d41700 1 Processor -- start 2023-07-14T17:36:59.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.108+0000 7f1ae1d41700 1 -- start start 2023-07-14T17:36:59.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.108+0000 7f1ae1d41700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc104660 0x7f1adc195850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:59.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.108+0000 7f1ae1d41700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1adc195d90 con 0x7f1adc104660 2023-07-14T17:36:59.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.108+0000 7f1adb7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc104660 0x7f1adc195850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:59.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.108+0000 7f1adb7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc104660 0x7f1adc195850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40460/0 (socket says 172.21.15.88:40460) 2023-07-14T17:36:59.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.108+0000 7f1adb7fe700 1 -- 172.21.15.88:0/1908500509 learned_addr learned my addr 172.21.15.88:0/1908500509 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:36:59.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.109+0000 7f1adb7fe700 1 -- 172.21.15.88:0/1908500509 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac40095f0 con 0x7f1adc104660 2023-07-14T17:36:59.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.109+0000 7f1adb7fe700 1 --2- 172.21.15.88:0/1908500509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc104660 0x7f1adc195850 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4004f40 tx=0x7f1ac4005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:59.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.109+0000 7f1ad8ff9700 1 -- 172.21.15.88:0/1908500509 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ac401d070 con 0x7f1adc104660 2023-07-14T17:36:59.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.109+0000 7f1ad8ff9700 1 -- 172.21.15.88:0/1908500509 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1ac40044b0 con 0x7f1adc104660 2023-07-14T17:36:59.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.109+0000 7f1ad8ff9700 1 -- 172.21.15.88:0/1908500509 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ac4003810 con 0x7f1adc104660 2023-07-14T17:36:59.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.109+0000 7f1ae1d41700 1 -- 172.21.15.88:0/1908500509 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1adc195ff0 con 0x7f1adc104660 2023-07-14T17:36:59.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.109+0000 7f1ae1d41700 1 -- 172.21.15.88:0/1908500509 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1adc1964f0 con 0x7f1adc104660 2023-07-14T17:36:59.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.110+0000 7f1ad8ff9700 1 -- 172.21.15.88:0/1908500509 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f1ac4003a90 con 0x7f1adc104660 2023-07-14T17:36:59.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.110+0000 7f1ad8ff9700 1 --2- 172.21.15.88:0/1908500509 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1ac80387a0 0x7f1ac803ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:36:59.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.110+0000 7f1ae1d41700 1 -- 172.21.15.88:0/1908500509 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1adc103900 con 0x7f1adc104660 2023-07-14T17:36:59.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.110+0000 7f1ad8ff9700 1 -- 172.21.15.88:0/1908500509 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1ac404c970 con 0x7f1adc104660 2023-07-14T17:36:59.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.110+0000 7f1adaffd700 1 --2- 172.21.15.88:0/1908500509 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1ac80387a0 0x7f1ac803ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:36:59.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.110+0000 7f1adaffd700 1 --2- 172.21.15.88:0/1908500509 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1ac80387a0 0x7f1ac803ac50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1acc006fd0 tx=0x7f1acc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:36:59.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.115+0000 7f1ad8ff9700 1 -- 172.21.15.88:0/1908500509 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1ac402a770 con 0x7f1adc104660 2023-07-14T17:36:59.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:58 smithi088 ceph-mon[127465]: Updating smithi088:/etc/ceph/ceph.client.admin.keyring 2023-07-14T17:36:59.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:36:58 smithi088 ceph-mon[127465]: Updating smithi088:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T17:36:59.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:36:59.304+0000 7f1ae1d41700 1 -- 172.21.15.88:0/1908500509 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi107", "target": ["mon-mgr", ""]}) v1 -- 0x7f1adc067f40 con 0x7f1ac80387a0 2023-07-14T17:37:00.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:00 smithi088 ceph-mon[127465]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi107", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:37:00.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:00 smithi088 ceph-mon[127465]: Deploying cephadm binary to smithi107 2023-07-14T17:37:01.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:01.575+0000 7f1ad8ff9700 1 -- 172.21.15.88:0/1908500509 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f1adc067f40 con 0x7f1ac80387a0 2023-07-14T17:37:01.675 INFO:teuthology.orchestra.run.smithi088.stdout:Added host 'smithi107' with addr '172.21.15.107' 2023-07-14T17:37:01.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:01.577+0000 7f1ae1d41700 1 -- 172.21.15.88:0/1908500509 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1ac80387a0 msgr2=0x7f1ac803ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:01.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:01.577+0000 7f1ae1d41700 1 --2- 172.21.15.88:0/1908500509 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1ac80387a0 0x7f1ac803ac50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1acc006fd0 tx=0x7f1acc006e40 comp rx=0 tx=0).stop 2023-07-14T17:37:01.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:01.577+0000 7f1ae1d41700 1 -- 172.21.15.88:0/1908500509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc104660 msgr2=0x7f1adc195850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:01.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:01.577+0000 7f1ae1d41700 1 --2- 172.21.15.88:0/1908500509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc104660 0x7f1adc195850 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4004f40 tx=0x7f1ac4005020 comp rx=0 tx=0).stop 2023-07-14T17:37:01.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:01.577+0000 7f1ae1d41700 1 -- 172.21.15.88:0/1908500509 shutdown_connections 2023-07-14T17:37:01.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:01.577+0000 7f1ae1d41700 1 --2- 172.21.15.88:0/1908500509 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1ac80387a0 0x7f1ac803ac50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:01.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:01.577+0000 7f1ae1d41700 1 --2- 172.21.15.88:0/1908500509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1adc104660 0x7f1adc195850 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:01.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:01.577+0000 7f1ae1d41700 1 -- 172.21.15.88:0/1908500509 >> 172.21.15.88:0/1908500509 conn(0x7f1adc0743e0 msgr2=0x7f1adc105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:01.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:01.577+0000 7f1ae1d41700 1 -- 172.21.15.88:0/1908500509 shutdown_connections 2023-07-14T17:37:01.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:01.577+0000 7f1ae1d41700 1 -- 172.21.15.88:0/1908500509 wait complete. 2023-07-14T17:37:01.943 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch host ls --format=json 2023-07-14T17:37:02.499 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:37:02.857 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:02 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:02.857 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:02 smithi088 ceph-mon[127465]: Added host smithi107 2023-07-14T17:37:02.857 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:02 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:37:02.858 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:02 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:03.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.709+0000 7ff1cf332700 1 -- 172.21.15.88:0/2373980734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c8104660 msgr2=0x7ff1c8104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:03.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.709+0000 7ff1cf332700 1 --2- 172.21.15.88:0/2373980734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c8104660 0x7ff1c8104a30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8009910 tx=0x7ff1b8009c20 comp rx=0 tx=0).stop 2023-07-14T17:37:03.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.709+0000 7ff1cf332700 1 -- 172.21.15.88:0/2373980734 shutdown_connections 2023-07-14T17:37:03.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.709+0000 7ff1cf332700 1 --2- 172.21.15.88:0/2373980734 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c8104660 0x7ff1c8104a30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:03.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.709+0000 7ff1cf332700 1 -- 172.21.15.88:0/2373980734 >> 172.21.15.88:0/2373980734 conn(0x7ff1c80743e0 msgr2=0x7ff1c80747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:03.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.709+0000 7ff1cf332700 1 -- 172.21.15.88:0/2373980734 shutdown_connections 2023-07-14T17:37:03.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.709+0000 7ff1cf332700 1 -- 172.21.15.88:0/2373980734 wait complete. 2023-07-14T17:37:03.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.710+0000 7ff1cf332700 1 Processor -- start 2023-07-14T17:37:03.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.710+0000 7ff1cf332700 1 -- start start 2023-07-14T17:37:03.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.710+0000 7ff1cf332700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c8104660 0x7ff1c819ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:03.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.710+0000 7ff1cf332700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c819cf60 con 0x7ff1c8104660 2023-07-14T17:37:03.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.710+0000 7ff1cd0ce700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c8104660 0x7ff1c819ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:03.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.710+0000 7ff1cd0ce700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c8104660 0x7ff1c819ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40486/0 (socket says 172.21.15.88:40486) 2023-07-14T17:37:03.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.710+0000 7ff1cd0ce700 1 -- 172.21.15.88:0/727134527 learned_addr learned my addr 172.21.15.88:0/727134527 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:37:03.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.711+0000 7ff1cd0ce700 1 -- 172.21.15.88:0/727134527 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b80095f0 con 0x7ff1c8104660 2023-07-14T17:37:03.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.711+0000 7ff1cd0ce700 1 --2- 172.21.15.88:0/727134527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c8104660 0x7ff1c819ca20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8006010 tx=0x7ff1b8004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:03.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.711+0000 7ff1be7fc700 1 -- 172.21.15.88:0/727134527 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1b801d070 con 0x7ff1c8104660 2023-07-14T17:37:03.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.711+0000 7ff1be7fc700 1 -- 172.21.15.88:0/727134527 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff1b800bdf0 con 0x7ff1c8104660 2023-07-14T17:37:03.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.711+0000 7ff1cf332700 1 -- 172.21.15.88:0/727134527 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c819d1c0 con 0x7ff1c8104660 2023-07-14T17:37:03.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.711+0000 7ff1be7fc700 1 -- 172.21.15.88:0/727134527 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1b8022d80 con 0x7ff1c8104660 2023-07-14T17:37:03.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.711+0000 7ff1cf332700 1 -- 172.21.15.88:0/727134527 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c8196a10 con 0x7ff1c8104660 2023-07-14T17:37:03.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.712+0000 7ff1be7fc700 1 -- 172.21.15.88:0/727134527 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7ff1b800f460 con 0x7ff1c8104660 2023-07-14T17:37:03.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.712+0000 7ff1cf332700 1 -- 172.21.15.88:0/727134527 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c8061c70 con 0x7ff1c8104660 2023-07-14T17:37:03.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.712+0000 7ff1be7fc700 1 --2- 172.21.15.88:0/727134527 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff1b40387b0 0x7ff1b403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:03.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.712+0000 7ff1be7fc700 1 -- 172.21.15.88:0/727134527 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff1b804c9f0 con 0x7ff1c8104660 2023-07-14T17:37:03.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.712+0000 7ff1cc8cd700 1 --2- 172.21.15.88:0/727134527 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff1b40387b0 0x7ff1b403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:03.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.712+0000 7ff1cc8cd700 1 --2- 172.21.15.88:0/727134527 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff1b40387b0 0x7ff1b403ac60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4006fd0 tx=0x7ff1c4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:03.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.717+0000 7ff1be7fc700 1 -- 172.21.15.88:0/727134527 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff1b8020370 con 0x7ff1c8104660 2023-07-14T17:37:03.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.910+0000 7ff1cf332700 1 -- 172.21.15.88:0/727134527 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff1c8197540 con 0x7ff1b40387b0 2023-07-14T17:37:03.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.911+0000 7ff1be7fc700 1 -- 172.21.15.88:0/727134527 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7ff1c8197540 con 0x7ff1b40387b0 2023-07-14T17:37:03.897 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:37:03.897 INFO:teuthology.orchestra.run.smithi088.stdout:[{"addr": "172.21.15.88", "hostname": "smithi088", "labels": [], "status": ""}, {"addr": "172.21.15.107", "hostname": "smithi107", "labels": [], "status": ""}] 2023-07-14T17:37:03.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.913+0000 7ff1cf332700 1 -- 172.21.15.88:0/727134527 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff1b40387b0 msgr2=0x7ff1b403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:03.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.913+0000 7ff1cf332700 1 --2- 172.21.15.88:0/727134527 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff1b40387b0 0x7ff1b403ac60 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff1c4006fd0 tx=0x7ff1c4006e40 comp rx=0 tx=0).stop 2023-07-14T17:37:03.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.913+0000 7ff1cf332700 1 -- 172.21.15.88:0/727134527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c8104660 msgr2=0x7ff1c819ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:03.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.913+0000 7ff1cf332700 1 --2- 172.21.15.88:0/727134527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c8104660 0x7ff1c819ca20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8006010 tx=0x7ff1b8004ff0 comp rx=0 tx=0).stop 2023-07-14T17:37:03.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.913+0000 7ff1cf332700 1 -- 172.21.15.88:0/727134527 shutdown_connections 2023-07-14T17:37:03.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.913+0000 7ff1cf332700 1 --2- 172.21.15.88:0/727134527 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff1b40387b0 0x7ff1b403ac60 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:03.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.913+0000 7ff1cf332700 1 --2- 172.21.15.88:0/727134527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1c8104660 0x7ff1c819ca20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:03.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.913+0000 7ff1cf332700 1 -- 172.21.15.88:0/727134527 >> 172.21.15.88:0/727134527 conn(0x7ff1c80743e0 msgr2=0x7ff1c80680d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:03.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.913+0000 7ff1cf332700 1 -- 172.21.15.88:0/727134527 shutdown_connections 2023-07-14T17:37:03.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:03.913+0000 7ff1cf332700 1 -- 172.21.15.88:0/727134527 wait complete. 2023-07-14T17:37:03.926 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:03 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:04.239 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi178 2023-07-14T17:37:04.240 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-07-14T17:37:04.240 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/etc/ceph/ceph.conf 2023-07-14T17:37:04.260 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-07-14T17:37:04.261 DEBUG:teuthology.orchestra.run.smithi178:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T17:37:04.320 INFO:tasks.cephadm:Adding host smithi178 to orchestrator... 2023-07-14T17:37:04.320 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch host add smithi178 2023-07-14T17:37:04.873 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:37:05.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:04 smithi088 ceph-mon[127465]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:05.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:04 smithi088 ceph-mon[127465]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:37:05.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:04 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:06.011 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.025+0000 7f9edf731700 1 -- 172.21.15.88:0/3504886156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed8104660 msgr2=0x7f9ed8104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:06.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.025+0000 7f9edf731700 1 --2- 172.21.15.88:0/3504886156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed8104660 0x7f9ed8104a30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009910 tx=0x7f9ec8009c20 comp rx=0 tx=0).stop 2023-07-14T17:37:06.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.025+0000 7f9edf731700 1 -- 172.21.15.88:0/3504886156 shutdown_connections 2023-07-14T17:37:06.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.025+0000 7f9edf731700 1 --2- 172.21.15.88:0/3504886156 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed8104660 0x7f9ed8104a30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:06.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.025+0000 7f9edf731700 1 -- 172.21.15.88:0/3504886156 >> 172.21.15.88:0/3504886156 conn(0x7f9ed80fa140 msgr2=0x7f9ed80fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:06.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.025+0000 7f9edf731700 1 -- 172.21.15.88:0/3504886156 shutdown_connections 2023-07-14T17:37:06.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.025+0000 7f9edf731700 1 -- 172.21.15.88:0/3504886156 wait complete. 2023-07-14T17:37:06.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.026+0000 7f9edf731700 1 Processor -- start 2023-07-14T17:37:06.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.026+0000 7f9edf731700 1 -- start start 2023-07-14T17:37:06.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.026+0000 7f9edf731700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed8104660 0x7f9ed8195840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:06.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.026+0000 7f9edf731700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed8195d80 con 0x7f9ed8104660 2023-07-14T17:37:06.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.026+0000 7f9edd4cd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed8104660 0x7f9ed8195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:06.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.026+0000 7f9edd4cd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed8104660 0x7f9ed8195840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40492/0 (socket says 172.21.15.88:40492) 2023-07-14T17:37:06.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.026+0000 7f9edd4cd700 1 -- 172.21.15.88:0/2349406445 learned_addr learned my addr 172.21.15.88:0/2349406445 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:37:06.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.026+0000 7f9edd4cd700 1 -- 172.21.15.88:0/2349406445 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec80095f0 con 0x7f9ed8104660 2023-07-14T17:37:06.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.026+0000 7f9edd4cd700 1 --2- 172.21.15.88:0/2349406445 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed8104660 0x7f9ed8195840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8006010 tx=0x7f9ec8004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:06.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.026+0000 7f9ece7fc700 1 -- 172.21.15.88:0/2349406445 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ec801d070 con 0x7f9ed8104660 2023-07-14T17:37:06.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.026+0000 7f9ece7fc700 1 -- 172.21.15.88:0/2349406445 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9ec800bdf0 con 0x7f9ed8104660 2023-07-14T17:37:06.015 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.026+0000 7f9edf731700 1 -- 172.21.15.88:0/2349406445 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed8195fe0 con 0x7f9ed8104660 2023-07-14T17:37:06.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.027+0000 7f9ece7fc700 1 -- 172.21.15.88:0/2349406445 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ec8022d80 con 0x7f9ed8104660 2023-07-14T17:37:06.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.027+0000 7f9edf731700 1 -- 172.21.15.88:0/2349406445 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed81964e0 con 0x7f9ed8104660 2023-07-14T17:37:06.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.027+0000 7f9ece7fc700 1 -- 172.21.15.88:0/2349406445 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f9ec800f460 con 0x7f9ed8104660 2023-07-14T17:37:06.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.027+0000 7f9ece7fc700 1 --2- 172.21.15.88:0/2349406445 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9ec4038850 0x7f9ec403ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:06.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.027+0000 7f9ece7fc700 1 -- 172.21.15.88:0/2349406445 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9ec804c8d0 con 0x7f9ed8104660 2023-07-14T17:37:06.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.027+0000 7f9edf731700 1 -- 172.21.15.88:0/2349406445 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed8103900 con 0x7f9ed8104660 2023-07-14T17:37:06.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.028+0000 7f9edcccc700 1 --2- 172.21.15.88:0/2349406445 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9ec4038850 0x7f9ec403ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:06.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.028+0000 7f9edcccc700 1 --2- 172.21.15.88:0/2349406445 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9ec4038850 0x7f9ec403ad00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4006fd0 tx=0x7f9ed4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:06.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.032+0000 7f9ece7fc700 1 -- 172.21.15.88:0/2349406445 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9ec8027030 con 0x7f9ed8104660 2023-07-14T17:37:06.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:06.221+0000 7f9edf731700 1 -- 172.21.15.88:0/2349406445 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi178", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ed80686e0 con 0x7f9ec4038850 2023-07-14T17:37:06.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:06 smithi088 ceph-mon[127465]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:06.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:06 smithi088 ceph-mon[127465]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi178", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:37:06.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:06 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:06.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:06 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:06.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:06 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:06.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:06 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:06.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:06 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:37:06.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:06 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:06.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:06 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:37:07.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:07 smithi088 ceph-mon[127465]: Updating smithi107:/etc/ceph/ceph.conf 2023-07-14T17:37:07.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:07 smithi088 ceph-mon[127465]: Deploying cephadm binary to smithi178 2023-07-14T17:37:07.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:07 smithi088 ceph-mon[127465]: Updating smithi107:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T17:37:07.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:07 smithi088 ceph-mon[127465]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2023-07-14T17:37:08.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:08.491+0000 7f9ece7fc700 1 -- 172.21.15.88:0/2349406445 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9ed80686e0 con 0x7f9ec4038850 2023-07-14T17:37:08.477 INFO:teuthology.orchestra.run.smithi088.stdout:Added host 'smithi178' with addr '172.21.15.178' 2023-07-14T17:37:08.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:08.493+0000 7f9edf731700 1 -- 172.21.15.88:0/2349406445 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9ec4038850 msgr2=0x7f9ec403ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:08.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:08.493+0000 7f9edf731700 1 --2- 172.21.15.88:0/2349406445 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9ec4038850 0x7f9ec403ad00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9ed4006fd0 tx=0x7f9ed4006e40 comp rx=0 tx=0).stop 2023-07-14T17:37:08.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:08.493+0000 7f9edf731700 1 -- 172.21.15.88:0/2349406445 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed8104660 msgr2=0x7f9ed8195840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:08.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:08.493+0000 7f9edf731700 1 --2- 172.21.15.88:0/2349406445 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed8104660 0x7f9ed8195840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8006010 tx=0x7f9ec8004ff0 comp rx=0 tx=0).stop 2023-07-14T17:37:08.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:08.493+0000 7f9edf731700 1 -- 172.21.15.88:0/2349406445 shutdown_connections 2023-07-14T17:37:08.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:08.493+0000 7f9edf731700 1 --2- 172.21.15.88:0/2349406445 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9ec4038850 0x7f9ec403ad00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:08.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:08.493+0000 7f9edf731700 1 --2- 172.21.15.88:0/2349406445 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ed8104660 0x7f9ed8195840 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:08.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:08.493+0000 7f9edf731700 1 -- 172.21.15.88:0/2349406445 >> 172.21.15.88:0/2349406445 conn(0x7f9ed80fa140 msgr2=0x7f9ed80fada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:08.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:08.493+0000 7f9edf731700 1 -- 172.21.15.88:0/2349406445 shutdown_connections 2023-07-14T17:37:08.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:08.493+0000 7f9edf731700 1 -- 172.21.15.88:0/2349406445 wait complete. 2023-07-14T17:37:08.819 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch host ls --format=json 2023-07-14T17:37:09.373 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:37:09.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:09 smithi088 ceph-mon[127465]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:09.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:09 smithi088 ceph-mon[127465]: Updating smithi107:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T17:37:09.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:09 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:09.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:09 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:09.447 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:09 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:09.447 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:09 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:09.447 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:09 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:37:09.447 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:09 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:10.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:10 smithi088 ceph-mon[127465]: Added host smithi178 2023-07-14T17:37:10.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:10 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:10.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.666+0000 7f880940b700 1 -- 172.21.15.88:0/394142460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8804104660 msgr2=0x7f8804104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:10.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.666+0000 7f880940b700 1 --2- 172.21.15.88:0/394142460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8804104660 0x7f8804104a30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f87f4009910 tx=0x7f87f4009c20 comp rx=0 tx=0).stop 2023-07-14T17:37:10.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.666+0000 7f880940b700 1 -- 172.21.15.88:0/394142460 shutdown_connections 2023-07-14T17:37:10.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.666+0000 7f880940b700 1 --2- 172.21.15.88:0/394142460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8804104660 0x7f8804104a30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:10.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.666+0000 7f880940b700 1 -- 172.21.15.88:0/394142460 >> 172.21.15.88:0/394142460 conn(0x7f88040743e0 msgr2=0x7f88040747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:10.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.666+0000 7f880940b700 1 -- 172.21.15.88:0/394142460 shutdown_connections 2023-07-14T17:37:10.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.667+0000 7f880940b700 1 -- 172.21.15.88:0/394142460 wait complete. 2023-07-14T17:37:10.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.667+0000 7f880940b700 1 Processor -- start 2023-07-14T17:37:10.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.667+0000 7f880940b700 1 -- start start 2023-07-14T17:37:10.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.667+0000 7f880940b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8804104660 0x7f8804193640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:10.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.667+0000 7f880940b700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8804068740 con 0x7f8804104660 2023-07-14T17:37:10.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.668+0000 7f8802ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8804104660 0x7f8804193640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:10.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.668+0000 7f8802ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8804104660 0x7f8804193640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40518/0 (socket says 172.21.15.88:40518) 2023-07-14T17:37:10.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.668+0000 7f8802ffd700 1 -- 172.21.15.88:0/4129589104 learned_addr learned my addr 172.21.15.88:0/4129589104 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:37:10.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.668+0000 7f8802ffd700 1 -- 172.21.15.88:0/4129589104 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f40095f0 con 0x7f8804104660 2023-07-14T17:37:10.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.668+0000 7f8802ffd700 1 --2- 172.21.15.88:0/4129589104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8804104660 0x7f8804193640 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f87f4004f40 tx=0x7f87f4005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:10.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.668+0000 7f87ebfff700 1 -- 172.21.15.88:0/4129589104 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87f401d070 con 0x7f8804104660 2023-07-14T17:37:10.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.668+0000 7f87ebfff700 1 -- 172.21.15.88:0/4129589104 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f87f40044b0 con 0x7f8804104660 2023-07-14T17:37:10.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.668+0000 7f880940b700 1 -- 172.21.15.88:0/4129589104 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8804193b80 con 0x7f8804104660 2023-07-14T17:37:10.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.668+0000 7f87ebfff700 1 -- 172.21.15.88:0/4129589104 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87f4003860 con 0x7f8804104660 2023-07-14T17:37:10.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.668+0000 7f880940b700 1 -- 172.21.15.88:0/4129589104 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8804194080 con 0x7f8804104660 2023-07-14T17:37:10.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.669+0000 7f87ebfff700 1 -- 172.21.15.88:0/4129589104 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f87f4003ae0 con 0x7f8804104660 2023-07-14T17:37:10.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.669+0000 7f87ebfff700 1 --2- 172.21.15.88:0/4129589104 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f87ec0387b0 0x7f87ec03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:10.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.669+0000 7f87ebfff700 1 -- 172.21.15.88:0/4129589104 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f87f404c850 con 0x7f8804104660 2023-07-14T17:37:10.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.669+0000 7f88027fc700 1 --2- 172.21.15.88:0/4129589104 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f87ec0387b0 0x7f87ec03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:10.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.669+0000 7f880940b700 1 -- 172.21.15.88:0/4129589104 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8804103900 con 0x7f8804104660 2023-07-14T17:37:10.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.670+0000 7f88027fc700 1 --2- 172.21.15.88:0/4129589104 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f87ec0387b0 0x7f87ec03ac60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f87f8006fd0 tx=0x7f87f8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:10.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.674+0000 7f87ebfff700 1 -- 172.21.15.88:0/4129589104 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f87f402a770 con 0x7f8804104660 2023-07-14T17:37:10.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.866+0000 7f880940b700 1 -- 172.21.15.88:0/4129589104 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f88041086c0 con 0x7f87ec0387b0 2023-07-14T17:37:10.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.868+0000 7f87ebfff700 1 -- 172.21.15.88:0/4129589104 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f88041086c0 con 0x7f87ec0387b0 2023-07-14T17:37:10.854 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:37:10.854 INFO:teuthology.orchestra.run.smithi088.stdout:[{"addr": "172.21.15.88", "hostname": "smithi088", "labels": [], "status": ""}, {"addr": "172.21.15.107", "hostname": "smithi107", "labels": [], "status": ""}, {"addr": "172.21.15.178", "hostname": "smithi178", "labels": [], "status": ""}] 2023-07-14T17:37:10.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.870+0000 7f880940b700 1 -- 172.21.15.88:0/4129589104 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f87ec0387b0 msgr2=0x7f87ec03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:10.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.870+0000 7f880940b700 1 --2- 172.21.15.88:0/4129589104 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f87ec0387b0 0x7f87ec03ac60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f87f8006fd0 tx=0x7f87f8006e40 comp rx=0 tx=0).stop 2023-07-14T17:37:10.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.870+0000 7f880940b700 1 -- 172.21.15.88:0/4129589104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8804104660 msgr2=0x7f8804193640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:10.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.870+0000 7f880940b700 1 --2- 172.21.15.88:0/4129589104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8804104660 0x7f8804193640 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f87f4004f40 tx=0x7f87f4005020 comp rx=0 tx=0).stop 2023-07-14T17:37:10.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.870+0000 7f880940b700 1 -- 172.21.15.88:0/4129589104 shutdown_connections 2023-07-14T17:37:10.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.870+0000 7f880940b700 1 --2- 172.21.15.88:0/4129589104 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f87ec0387b0 0x7f87ec03ac60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:10.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.870+0000 7f880940b700 1 --2- 172.21.15.88:0/4129589104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8804104660 0x7f8804193640 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:10.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.870+0000 7f880940b700 1 -- 172.21.15.88:0/4129589104 >> 172.21.15.88:0/4129589104 conn(0x7f88040743e0 msgr2=0x7f8804105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:10.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.870+0000 7f880940b700 1 -- 172.21.15.88:0/4129589104 shutdown_connections 2023-07-14T17:37:10.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:10.870+0000 7f880940b700 1 -- 172.21.15.88:0/4129589104 wait complete. 2023-07-14T17:37:11.227 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-14T17:37:11.228 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd crush tunables default 2023-07-14T17:37:11.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:11 smithi088 ceph-mon[127465]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:11.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:11 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:11.776 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:37:12.279 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:12 smithi088 ceph-mon[127465]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:37:12.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.979+0000 7f1ee9367700 1 -- 172.21.15.88:0/2202160243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ee40fe770 msgr2=0x7f1ee40feb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:12.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.979+0000 7f1ee9367700 1 --2- 172.21.15.88:0/2202160243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ee40fe770 0x7f1ee40feb40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8009910 tx=0x7f1ed8009c20 comp rx=0 tx=0).stop 2023-07-14T17:37:12.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.979+0000 7f1ee9367700 1 -- 172.21.15.88:0/2202160243 shutdown_connections 2023-07-14T17:37:12.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.979+0000 7f1ee9367700 1 --2- 172.21.15.88:0/2202160243 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ee40fe770 0x7f1ee40feb40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:12.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.979+0000 7f1ee9367700 1 -- 172.21.15.88:0/2202160243 >> 172.21.15.88:0/2202160243 conn(0x7f1ee4074e40 msgr2=0x7f1ee4075240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:12.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.979+0000 7f1ee9367700 1 -- 172.21.15.88:0/2202160243 shutdown_connections 2023-07-14T17:37:12.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.979+0000 7f1ee9367700 1 -- 172.21.15.88:0/2202160243 wait complete. 2023-07-14T17:37:12.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.979+0000 7f1ee9367700 1 Processor -- start 2023-07-14T17:37:12.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.979+0000 7f1ee9367700 1 -- start start 2023-07-14T17:37:12.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.980+0000 7f1ee9367700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ee40fe770 0x7f1ee419a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:12.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.980+0000 7f1ee9367700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee419aec0 con 0x7f1ee40fe770 2023-07-14T17:37:12.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.980+0000 7f1ee2ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ee40fe770 0x7f1ee419a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:12.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.980+0000 7f1ee2ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ee40fe770 0x7f1ee419a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40524/0 (socket says 172.21.15.88:40524) 2023-07-14T17:37:12.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.980+0000 7f1ee2ffd700 1 -- 172.21.15.88:0/3442476058 learned_addr learned my addr 172.21.15.88:0/3442476058 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:37:12.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.980+0000 7f1ee2ffd700 1 -- 172.21.15.88:0/3442476058 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed80095f0 con 0x7f1ee40fe770 2023-07-14T17:37:12.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.980+0000 7f1ee2ffd700 1 --2- 172.21.15.88:0/3442476058 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ee40fe770 0x7f1ee419a980 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8004f40 tx=0x7f1ed8005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:12.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.980+0000 7f1ecbfff700 1 -- 172.21.15.88:0/3442476058 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ed801d070 con 0x7f1ee40fe770 2023-07-14T17:37:12.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.980+0000 7f1ecbfff700 1 -- 172.21.15.88:0/3442476058 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1ed80044b0 con 0x7f1ee40fe770 2023-07-14T17:37:12.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.980+0000 7f1ee9367700 1 -- 172.21.15.88:0/3442476058 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee419b0c0 con 0x7f1ee40fe770 2023-07-14T17:37:12.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.980+0000 7f1ecbfff700 1 -- 172.21.15.88:0/3442476058 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1ed8003860 con 0x7f1ee40fe770 2023-07-14T17:37:12.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.980+0000 7f1ee9367700 1 -- 172.21.15.88:0/3442476058 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee419b480 con 0x7f1ee40fe770 2023-07-14T17:37:12.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.981+0000 7f1ecbfff700 1 -- 172.21.15.88:0/3442476058 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f1ed8003a90 con 0x7f1ee40fe770 2023-07-14T17:37:12.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.981+0000 7f1ecbfff700 1 --2- 172.21.15.88:0/3442476058 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1ecc038800 0x7f1ecc03acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:12.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.981+0000 7f1ecbfff700 1 -- 172.21.15.88:0/3442476058 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1ed804c750 con 0x7f1ee40fe770 2023-07-14T17:37:12.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.981+0000 7f1ee27fc700 1 --2- 172.21.15.88:0/3442476058 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1ecc038800 0x7f1ecc03acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:12.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.981+0000 7f1ee9367700 1 -- 172.21.15.88:0/3442476058 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee4061c70 con 0x7f1ee40fe770 2023-07-14T17:37:12.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.982+0000 7f1ee27fc700 1 --2- 172.21.15.88:0/3442476058 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1ecc038800 0x7f1ecc03acb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4006fd0 tx=0x7f1ed4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:12.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:12.986+0000 7f1ecbfff700 1 -- 172.21.15.88:0/3442476058 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1ed802a770 con 0x7f1ee40fe770 2023-07-14T17:37:13.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:13.174+0000 7f1ee9367700 1 -- 172.21.15.88:0/3442476058 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f1ee4195f10 con 0x7f1ee40fe770 2023-07-14T17:37:13.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:13 smithi088 ceph-mon[127465]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:14.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:14.178+0000 7f1ecbfff700 1 -- 172.21.15.88:0/3442476058 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1ed8020370 con 0x7f1ee40fe770 2023-07-14T17:37:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:14.181+0000 7f1ee9367700 1 -- 172.21.15.88:0/3442476058 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1ecc038800 msgr2=0x7f1ecc03acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:14.181+0000 7f1ee9367700 1 --2- 172.21.15.88:0/3442476058 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1ecc038800 0x7f1ecc03acb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4006fd0 tx=0x7f1ed4006e40 comp rx=0 tx=0).stop 2023-07-14T17:37:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:14.181+0000 7f1ee9367700 1 -- 172.21.15.88:0/3442476058 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ee40fe770 msgr2=0x7f1ee419a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:14.181+0000 7f1ee9367700 1 --2- 172.21.15.88:0/3442476058 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ee40fe770 0x7f1ee419a980 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1ed8004f40 tx=0x7f1ed8005020 comp rx=0 tx=0).stop 2023-07-14T17:37:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:14.181+0000 7f1ee9367700 1 -- 172.21.15.88:0/3442476058 shutdown_connections 2023-07-14T17:37:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:14.181+0000 7f1ee9367700 1 --2- 172.21.15.88:0/3442476058 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1ecc038800 0x7f1ecc03acb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:14.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:14.181+0000 7f1ee9367700 1 --2- 172.21.15.88:0/3442476058 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ee40fe770 0x7f1ee419a980 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:14.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:14.181+0000 7f1ee9367700 1 -- 172.21.15.88:0/3442476058 >> 172.21.15.88:0/3442476058 conn(0x7f1ee4074e40 msgr2=0x7f1ee4104330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:14.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:14.181+0000 7f1ee9367700 1 -- 172.21.15.88:0/3442476058 shutdown_connections 2023-07-14T17:37:14.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:14.181+0000 7f1ee9367700 1 -- 172.21.15.88:0/3442476058 wait complete. 2023-07-14T17:37:14.168 INFO:teuthology.orchestra.run.smithi088.stderr:adjusted tunables profile to default 2023-07-14T17:37:14.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:14 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3442476058' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T17:37:14.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:14.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:14.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:14.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:14.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi178", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:37:14.447 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:14.447 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:37:14.533 INFO:tasks.cephadm:Adding mon.a on smithi088 2023-07-14T17:37:14.533 INFO:tasks.cephadm:Adding mon.b on smithi107 2023-07-14T17:37:14.533 INFO:tasks.cephadm:Adding mon.c on smithi178 2023-07-14T17:37:14.534 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch apply mon '3;smithi088:172.21.15.88=a;smithi107:172.21.15.107=b;smithi178:172.21.15.178=c' 2023-07-14T17:37:15.103 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T17:37:15.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:15 smithi088 ceph-mon[127465]: Updating smithi178:/etc/ceph/ceph.conf 2023-07-14T17:37:15.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:15 smithi088 ceph-mon[127465]: Updating smithi178:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T17:37:15.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:15 smithi088 ceph-mon[127465]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:15.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:15 smithi088 ceph-mon[127465]: Updating smithi178:/etc/ceph/ceph.client.admin.keyring 2023-07-14T17:37:15.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:15 smithi088 ceph-mon[127465]: Updating smithi178:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T17:37:15.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:15 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3442476058' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T17:37:15.447 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:15 smithi088 ceph-mon[127465]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T17:37:15.447 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:15 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:15.447 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:15 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:15.447 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:15 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:16.333 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.346+0000 7ff7cf788700 1 -- 172.21.15.178:0/3663497029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7c80fe580 msgr2=0x7ff7c80fe950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:16.333 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.346+0000 7ff7cf788700 1 --2- 172.21.15.178:0/3663497029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7c80fe580 0x7ff7c80fe950 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8009910 tx=0x7ff7b8009c20 comp rx=0 tx=0).stop 2023-07-14T17:37:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.346+0000 7ff7cf788700 1 -- 172.21.15.178:0/3663497029 shutdown_connections 2023-07-14T17:37:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.346+0000 7ff7cf788700 1 --2- 172.21.15.178:0/3663497029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7c80fe580 0x7ff7c80fe950 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.346+0000 7ff7cf788700 1 -- 172.21.15.178:0/3663497029 >> 172.21.15.178:0/3663497029 conn(0x7ff7c8075040 msgr2=0x7ff7c8075440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.346+0000 7ff7cf788700 1 -- 172.21.15.178:0/3663497029 shutdown_connections 2023-07-14T17:37:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.346+0000 7ff7cf788700 1 -- 172.21.15.178:0/3663497029 wait complete. 2023-07-14T17:37:16.334 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.346+0000 7ff7cf788700 1 Processor -- start 2023-07-14T17:37:16.335 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.346+0000 7ff7cf788700 1 -- start start 2023-07-14T17:37:16.335 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.347+0000 7ff7cf788700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7c80fe580 0x7ff7c80ff500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:16.335 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.347+0000 7ff7cf788700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c8101480 con 0x7ff7c80fe580 2023-07-14T17:37:16.335 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.347+0000 7ff7cd524700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7c80fe580 0x7ff7c80ff500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:16.335 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.347+0000 7ff7cd524700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7c80fe580 0x7ff7c80ff500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.178:51376/0 (socket says 172.21.15.178:51376) 2023-07-14T17:37:16.335 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.347+0000 7ff7cd524700 1 -- 172.21.15.178:0/4156821176 learned_addr learned my addr 172.21.15.178:0/4156821176 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-07-14T17:37:16.336 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.347+0000 7ff7cd524700 1 -- 172.21.15.178:0/4156821176 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b80095f0 con 0x7ff7c80fe580 2023-07-14T17:37:16.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.347+0000 7ff7cd524700 1 --2- 172.21.15.178:0/4156821176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7c80fe580 0x7ff7c80ff500 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8004f40 tx=0x7ff7b8005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:16.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.348+0000 7ff7be7fc700 1 -- 172.21.15.178:0/4156821176 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7b801d070 con 0x7ff7c80fe580 2023-07-14T17:37:16.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.348+0000 7ff7cf788700 1 -- 172.21.15.178:0/4156821176 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c80ffa40 con 0x7ff7c80fe580 2023-07-14T17:37:16.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.348+0000 7ff7be7fc700 1 -- 172.21.15.178:0/4156821176 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff7b80044b0 con 0x7ff7c80fe580 2023-07-14T17:37:16.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.348+0000 7ff7be7fc700 1 -- 172.21.15.178:0/4156821176 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7b8003860 con 0x7ff7c80fe580 2023-07-14T17:37:16.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.348+0000 7ff7cf788700 1 -- 172.21.15.178:0/4156821176 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c80ffec0 con 0x7ff7c80fe580 2023-07-14T17:37:16.337 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.349+0000 7ff7be7fc700 1 -- 172.21.15.178:0/4156821176 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7ff7b800b960 con 0x7ff7c80fe580 2023-07-14T17:37:16.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.349+0000 7ff7cf788700 1 -- 172.21.15.178:0/4156821176 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c8106da0 con 0x7ff7c80fe580 2023-07-14T17:37:16.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.349+0000 7ff7be7fc700 1 --2- 172.21.15.178:0/4156821176 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff7b40387b0 0x7ff7b403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:16.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.349+0000 7ff7be7fc700 1 -- 172.21.15.178:0/4156821176 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff7b804c630 con 0x7ff7c80fe580 2023-07-14T17:37:16.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.349+0000 7ff7ccd23700 1 --2- 172.21.15.178:0/4156821176 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff7b40387b0 0x7ff7b403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:16.338 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.350+0000 7ff7ccd23700 1 --2- 172.21.15.178:0/4156821176 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff7b40387b0 0x7ff7b403ac60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4006fd0 tx=0x7ff7c4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:16.340 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.354+0000 7ff7be7fc700 1 -- 172.21.15.178:0/4156821176 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff7b802a540 con 0x7ff7c80fe580 2023-07-14T17:37:16.533 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.547+0000 7ff7cf788700 1 -- 172.21.15.178:0/4156821176 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi088:172.21.15.88=a;smithi107:172.21.15.107=b;smithi178:172.21.15.178=c", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7c8100290 con 0x7ff7b40387b0 2023-07-14T17:37:16.540 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.554+0000 7ff7be7fc700 1 -- 172.21.15.178:0/4156821176 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff7c8100290 con 0x7ff7b40387b0 2023-07-14T17:37:16.541 INFO:teuthology.orchestra.run.smithi178.stdout:Scheduled mon update... 2023-07-14T17:37:16.542 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.556+0000 7ff7cf788700 1 -- 172.21.15.178:0/4156821176 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff7b40387b0 msgr2=0x7ff7b403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:16.542 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.556+0000 7ff7cf788700 1 --2- 172.21.15.178:0/4156821176 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff7b40387b0 0x7ff7b403ac60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4006fd0 tx=0x7ff7c4006e40 comp rx=0 tx=0).stop 2023-07-14T17:37:16.542 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.556+0000 7ff7cf788700 1 -- 172.21.15.178:0/4156821176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7c80fe580 msgr2=0x7ff7c80ff500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:16.542 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.556+0000 7ff7cf788700 1 --2- 172.21.15.178:0/4156821176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7c80fe580 0x7ff7c80ff500 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff7b8004f40 tx=0x7ff7b8005020 comp rx=0 tx=0).stop 2023-07-14T17:37:16.543 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.556+0000 7ff7cf788700 1 -- 172.21.15.178:0/4156821176 shutdown_connections 2023-07-14T17:37:16.543 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.556+0000 7ff7cf788700 1 --2- 172.21.15.178:0/4156821176 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff7b40387b0 0x7ff7b403ac60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:16.543 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.556+0000 7ff7cf788700 1 --2- 172.21.15.178:0/4156821176 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff7c80fe580 0x7ff7c80ff500 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:16.543 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.556+0000 7ff7cf788700 1 -- 172.21.15.178:0/4156821176 >> 172.21.15.178:0/4156821176 conn(0x7ff7c8075040 msgr2=0x7ff7c8103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:16.543 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.556+0000 7ff7cf788700 1 -- 172.21.15.178:0/4156821176 shutdown_connections 2023-07-14T17:37:16.543 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:16.556+0000 7ff7cf788700 1 -- 172.21.15.178:0/4156821176 wait complete. 2023-07-14T17:37:16.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:16 smithi088 ceph-mon[127465]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:16.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:16 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:16.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:16 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:37:16.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:16 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:16.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:16 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:37:16.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:16 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:16.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:16 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T17:37:16.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:16 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:16.960 DEBUG:teuthology.orchestra.run.smithi107:mon.b> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mon.b.service 2023-07-14T17:37:16.963 DEBUG:teuthology.orchestra.run.smithi178:mon.c> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mon.c.service 2023-07-14T17:37:16.967 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T17:37:16.968 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T17:37:16.997 INFO:journalctl@ceph.mon.c.smithi178.stdout:-- Logs begin at Fri 2023-07-14 17:14:18 UTC. -- 2023-07-14T17:37:16.999 INFO:journalctl@ceph.mon.b.smithi107.stdout:-- Logs begin at Fri 2023-07-14 17:14:00 UTC. -- 2023-07-14T17:37:17.845 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T17:37:17.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:17 smithi088 ceph-mon[127465]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi088:172.21.15.88=a;smithi107:172.21.15.107=b;smithi178:172.21.15.178=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:37:17.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:17 smithi088 ceph-mon[127465]: Saving service mon spec with placement smithi088:172.21.15.88=a;smithi107:172.21.15.107=b;smithi178:172.21.15.178=c;count:3 2023-07-14T17:37:17.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:17 smithi088 ceph-mon[127465]: Deploying daemon mon.c on smithi178 2023-07-14T17:37:18.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:18 smithi088 ceph-mon[127465]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:19.633 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.646+0000 7f90db110700 1 -- 172.21.15.178:0/906219164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d41057c0 msgr2=0x7f90d4105b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:19.633 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.646+0000 7f90db110700 1 --2- 172.21.15.178:0/906219164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d41057c0 0x7f90d4105b90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009910 tx=0x7f90c8009c20 comp rx=0 tx=0).stop 2023-07-14T17:37:19.634 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.646+0000 7f90db110700 1 -- 172.21.15.178:0/906219164 shutdown_connections 2023-07-14T17:37:19.634 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.646+0000 7f90db110700 1 --2- 172.21.15.178:0/906219164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d41057c0 0x7f90d4105b90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:19.634 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.646+0000 7f90db110700 1 -- 172.21.15.178:0/906219164 >> 172.21.15.178:0/906219164 conn(0x7f90d40743e0 msgr2=0x7f90d40747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:19.634 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.646+0000 7f90db110700 1 -- 172.21.15.178:0/906219164 shutdown_connections 2023-07-14T17:37:19.634 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.646+0000 7f90db110700 1 -- 172.21.15.178:0/906219164 wait complete. 2023-07-14T17:37:19.634 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.646+0000 7f90db110700 1 Processor -- start 2023-07-14T17:37:19.634 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.647+0000 7f90db110700 1 -- start start 2023-07-14T17:37:19.635 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.647+0000 7f90db110700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d41057c0 0x7f90d419a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:19.635 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.647+0000 7f90db110700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d419ade0 con 0x7f90d41057c0 2023-07-14T17:37:19.635 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.647+0000 7f90d8eac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d41057c0 0x7f90d419a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:19.635 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.647+0000 7f90d8eac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d41057c0 0x7f90d419a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.178:51382/0 (socket says 172.21.15.178:51382) 2023-07-14T17:37:19.635 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.647+0000 7f90d8eac700 1 -- 172.21.15.178:0/2804927830 learned_addr learned my addr 172.21.15.178:0/2804927830 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-07-14T17:37:19.636 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.647+0000 7f90d8eac700 1 -- 172.21.15.178:0/2804927830 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c80095f0 con 0x7f90d41057c0 2023-07-14T17:37:19.637 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.648+0000 7f90d8eac700 1 --2- 172.21.15.178:0/2804927830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d41057c0 0x7f90d419a8a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f90c8004f40 tx=0x7f90c8005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:19.637 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.648+0000 7f90d1ffb700 1 -- 172.21.15.178:0/2804927830 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90c801d070 con 0x7f90d41057c0 2023-07-14T17:37:19.637 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.648+0000 7f90db110700 1 -- 172.21.15.178:0/2804927830 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d419afe0 con 0x7f90d41057c0 2023-07-14T17:37:19.637 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.648+0000 7f90d1ffb700 1 -- 172.21.15.178:0/2804927830 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f90c80044b0 con 0x7f90d41057c0 2023-07-14T17:37:19.637 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.648+0000 7f90d1ffb700 1 -- 172.21.15.178:0/2804927830 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90c8003860 con 0x7f90d41057c0 2023-07-14T17:37:19.637 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.648+0000 7f90db110700 1 -- 172.21.15.178:0/2804927830 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d419b420 con 0x7f90d41057c0 2023-07-14T17:37:19.638 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.649+0000 7f90d1ffb700 1 -- 172.21.15.178:0/2804927830 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f90c8003ae0 con 0x7f90d41057c0 2023-07-14T17:37:19.638 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.649+0000 7f90db110700 1 -- 172.21.15.178:0/2804927830 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d4061c70 con 0x7f90d41057c0 2023-07-14T17:37:19.638 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.649+0000 7f90d1ffb700 1 --2- 172.21.15.178:0/2804927830 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f90c40387b0 0x7f90c403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:19.638 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.649+0000 7f90d1ffb700 1 -- 172.21.15.178:0/2804927830 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f90c804c890 con 0x7f90d41057c0 2023-07-14T17:37:19.638 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.650+0000 7f90d3fff700 1 --2- 172.21.15.178:0/2804927830 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f90c40387b0 0x7f90c403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:19.638 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.650+0000 7f90d3fff700 1 --2- 172.21.15.178:0/2804927830 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f90c40387b0 0x7f90c403ac60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f90c0006fd0 tx=0x7f90c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:19.641 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.655+0000 7f90d1ffb700 1 -- 172.21.15.178:0/2804927830 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f90c802a770 con 0x7f90d41057c0 2023-07-14T17:37:19.911 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.924+0000 7f90db110700 1 -- 172.21.15.178:0/2804927830 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f90d4195e30 con 0x7f90d41057c0 2023-07-14T17:37:19.911 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-07-14T17:37:19.912 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":1,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","modified":"2023-07-14T17:35:46.715826Z","created":"2023-07-14T17:35:46.715826Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-14T17:37:19.912 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.925+0000 7f90d1ffb700 1 -- 172.21.15.178:0/2804927830 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7f90c800e3e0 con 0x7f90d41057c0 2023-07-14T17:37:19.913 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.927+0000 7f90db110700 1 -- 172.21.15.178:0/2804927830 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f90c40387b0 msgr2=0x7f90c403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:19.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.927+0000 7f90db110700 1 --2- 172.21.15.178:0/2804927830 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f90c40387b0 0x7f90c403ac60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f90c0006fd0 tx=0x7f90c0006e40 comp rx=0 tx=0).stop 2023-07-14T17:37:19.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.927+0000 7f90db110700 1 -- 172.21.15.178:0/2804927830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d41057c0 msgr2=0x7f90d419a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:19.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.927+0000 7f90db110700 1 --2- 172.21.15.178:0/2804927830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d41057c0 0x7f90d419a8a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f90c8004f40 tx=0x7f90c8005020 comp rx=0 tx=0).stop 2023-07-14T17:37:19.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.927+0000 7f90db110700 1 -- 172.21.15.178:0/2804927830 shutdown_connections 2023-07-14T17:37:19.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.927+0000 7f90db110700 1 --2- 172.21.15.178:0/2804927830 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f90c40387b0 0x7f90c403ac60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:19.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.927+0000 7f90db110700 1 --2- 172.21.15.178:0/2804927830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f90d41057c0 0x7f90d419a8a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:19.914 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.927+0000 7f90db110700 1 -- 172.21.15.178:0/2804927830 >> 172.21.15.178:0/2804927830 conn(0x7f90d40743e0 msgr2=0x7f90d4106d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:19.915 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.927+0000 7f90db110700 1 -- 172.21.15.178:0/2804927830 shutdown_connections 2023-07-14T17:37:19.915 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:19.927+0000 7f90db110700 1 -- 172.21.15.178:0/2804927830 wait complete. 2023-07-14T17:37:19.915 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 1 2023-07-14T17:37:20.101 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:19 smithi178 podman[129304]: 2023-07-14T17:37:20.389 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 bash[129304]: f46f8e6951c92f3ae625abf3b8bcedcaa682c2fac4e4bacf5d4c57940f26097d 2023-07-14T17:37:20.389 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T17:37:20.390 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T17:37:20.390 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: pidfile_write: ignore empty --pid-file 2023-07-14T17:37:20.390 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: load: jerasure load: lrc 2023-07-14T17:37:20.390 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: RocksDB version: 7.9.2 2023-07-14T17:37:20.390 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Git sha 0 2023-07-14T17:37:20.390 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T17:37:20.390 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: DB SUMMARY 2023-07-14T17:37:20.391 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: DB Session ID: 8SISJ24HN2C8VXEW7E19 2023-07-14T17:37:20.391 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: CURRENT file: CURRENT 2023-07-14T17:37:20.391 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T17:37:20.391 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-14T17:37:20.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-07-14T17:37:20.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-07-14T17:37:20.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.error_if_exists: 0 2023-07-14T17:37:20.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.create_if_missing: 0 2023-07-14T17:37:20.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.paranoid_checks: 1 2023-07-14T17:37:20.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T17:37:20.392 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T17:37:20.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T17:37:20.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.env: 0x564c09076c60 2023-07-14T17:37:20.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T17:37:20.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.info_log: 0x564c0a9fd2a0 2023-07-14T17:37:20.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T17:37:20.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.statistics: (nil) 2023-07-14T17:37:20.393 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.use_fsync: 0 2023-07-14T17:37:20.394 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_log_file_size: 0 2023-07-14T17:37:20.394 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T17:37:20.394 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T17:37:20.394 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T17:37:20.394 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T17:37:20.394 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.allow_fallocate: 1 2023-07-14T17:37:20.394 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T17:37:20.394 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T17:37:20.395 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.use_direct_reads: 0 2023-07-14T17:37:20.395 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T17:37:20.395 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T17:37:20.395 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.db_log_dir: 2023-07-14T17:37:20.395 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.wal_dir: 2023-07-14T17:37:20.395 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T17:37:20.395 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T17:37:20.395 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T17:37:20.396 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T17:37:20.396 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T17:37:20.396 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T17:37:20.396 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T17:37:20.396 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T17:37:20.396 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.write_buffer_manager: 0x564c0a91e3c0 2023-07-14T17:37:20.396 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T17:37:20.397 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T17:37:20.397 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T17:37:20.397 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T17:37:20.397 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T17:37:20.397 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T17:37:20.397 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T17:37:20.398 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T17:37:20.398 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.unordered_write: 0 2023-07-14T17:37:20.398 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T17:37:20.398 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T17:37:20.399 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T17:37:20.399 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T17:37:20.399 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.row_cache: None 2023-07-14T17:37:20.400 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.wal_filter: None 2023-07-14T17:37:20.400 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T17:37:20.400 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T17:37:20.400 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.two_write_queues: 0 2023-07-14T17:37:20.401 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T17:37:20.401 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.wal_compression: 0 2023-07-14T17:37:20.401 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.atomic_flush: 0 2023-07-14T17:37:20.401 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T17:37:20.402 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T17:37:20.402 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T17:37:20.402 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.log_readahead_size: 0 2023-07-14T17:37:20.403 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T17:37:20.403 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T17:37:20.403 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T17:37:20.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T17:37:20.404 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T17:37:20.405 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T17:37:20.405 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T17:37:20.405 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_background_jobs: 2 2023-07-14T17:37:20.406 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_background_compactions: -1 2023-07-14T17:37:20.406 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_subcompactions: 1 2023-07-14T17:37:20.406 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T17:37:20.406 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T17:37:20.406 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T17:37:20.406 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T17:37:20.406 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T17:37:20.407 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T17:37:20.407 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T17:37:20.407 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T17:37:20.407 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_open_files: -1 2023-07-14T17:37:20.407 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T17:37:20.407 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T17:37:20.407 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T17:37:20.407 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T17:37:20.408 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_background_flushes: -1 2023-07-14T17:37:20.408 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Compression algorithms supported: 2023-07-14T17:37:20.408 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T17:37:20.408 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: kZSTD supported: 0 2023-07-14T17:37:20.408 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: kXpressCompression supported: 0 2023-07-14T17:37:20.408 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T17:37:20.408 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: kZlibCompression supported: 1 2023-07-14T17:37:20.409 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: kSnappyCompression supported: 1 2023-07-14T17:37:20.409 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: kLZ4Compression supported: 1 2023-07-14T17:37:20.409 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: kBZip2Compression supported: 0 2023-07-14T17:37:20.409 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T17:37:20.409 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T17:37:20.409 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-07-14T17:37:20.409 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T17:37:20.410 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T17:37:20.410 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.merge_operator: 2023-07-14T17:37:20.410 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compaction_filter: None 2023-07-14T17:37:20.410 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T17:37:20.410 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T17:37:20.410 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T17:37:20.410 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T17:37:20.410 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564c0a9fd3a0) 2023-07-14T17:37:20.411 INFO:journalctl@ceph.mon.c.smithi178.stdout: cache_index_and_filter_blocks: 1 2023-07-14T17:37:20.411 INFO:journalctl@ceph.mon.c.smithi178.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T17:37:20.411 INFO:journalctl@ceph.mon.c.smithi178.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T17:37:20.411 INFO:journalctl@ceph.mon.c.smithi178.stdout: pin_top_level_index_and_filter: 1 2023-07-14T17:37:20.411 INFO:journalctl@ceph.mon.c.smithi178.stdout: index_type: 0 2023-07-14T17:37:20.411 INFO:journalctl@ceph.mon.c.smithi178.stdout: data_block_index_type: 0 2023-07-14T17:37:20.411 INFO:journalctl@ceph.mon.c.smithi178.stdout: index_shortening: 1 2023-07-14T17:37:20.411 INFO:journalctl@ceph.mon.c.smithi178.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T17:37:20.412 INFO:journalctl@ceph.mon.c.smithi178.stdout: checksum: 4 2023-07-14T17:37:20.412 INFO:journalctl@ceph.mon.c.smithi178.stdout: no_block_cache: 0 2023-07-14T17:37:20.412 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_cache: 0x564c0a9bd1f0 2023-07-14T17:37:20.412 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_cache_name: BinnedLRUCache 2023-07-14T17:37:20.412 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_cache_options: 2023-07-14T17:37:20.412 INFO:journalctl@ceph.mon.c.smithi178.stdout: capacity : 536870912 2023-07-14T17:37:20.412 INFO:journalctl@ceph.mon.c.smithi178.stdout: num_shard_bits : 4 2023-07-14T17:37:20.413 INFO:journalctl@ceph.mon.c.smithi178.stdout: strict_capacity_limit : 0 2023-07-14T17:37:20.413 INFO:journalctl@ceph.mon.c.smithi178.stdout: high_pri_pool_ratio: 0.000 2023-07-14T17:37:20.413 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_cache_compressed: (nil) 2023-07-14T17:37:20.413 INFO:journalctl@ceph.mon.c.smithi178.stdout: persistent_cache: (nil) 2023-07-14T17:37:20.413 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_size: 4096 2023-07-14T17:37:20.413 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_size_deviation: 10 2023-07-14T17:37:20.413 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_restart_interval: 16 2023-07-14T17:37:20.414 INFO:journalctl@ceph.mon.c.smithi178.stdout: index_block_restart_interval: 1 2023-07-14T17:37:20.414 INFO:journalctl@ceph.mon.c.smithi178.stdout: metadata_block_size: 4096 2023-07-14T17:37:20.414 INFO:journalctl@ceph.mon.c.smithi178.stdout: partition_filters: 0 2023-07-14T17:37:20.414 INFO:journalctl@ceph.mon.c.smithi178.stdout: use_delta_encoding: 1 2023-07-14T17:37:20.414 INFO:journalctl@ceph.mon.c.smithi178.stdout: filter_policy: bloomfilter 2023-07-14T17:37:20.414 INFO:journalctl@ceph.mon.c.smithi178.stdout: whole_key_filtering: 1 2023-07-14T17:37:20.414 INFO:journalctl@ceph.mon.c.smithi178.stdout: verify_compression: 0 2023-07-14T17:37:20.414 INFO:journalctl@ceph.mon.c.smithi178.stdout: read_amp_bytes_per_bit: 0 2023-07-14T17:37:20.415 INFO:journalctl@ceph.mon.c.smithi178.stdout: format_version: 5 2023-07-14T17:37:20.415 INFO:journalctl@ceph.mon.c.smithi178.stdout: enable_index_compression: 1 2023-07-14T17:37:20.415 INFO:journalctl@ceph.mon.c.smithi178.stdout: block_align: 0 2023-07-14T17:37:20.415 INFO:journalctl@ceph.mon.c.smithi178.stdout: max_auto_readahead_size: 262144 2023-07-14T17:37:20.415 INFO:journalctl@ceph.mon.c.smithi178.stdout: prepopulate_block_cache: 0 2023-07-14T17:37:20.415 INFO:journalctl@ceph.mon.c.smithi178.stdout: initial_auto_readahead_size: 8192 2023-07-14T17:37:20.415 INFO:journalctl@ceph.mon.c.smithi178.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T17:37:20.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T17:37:20.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T17:37:20.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compression: NoCompression 2023-07-14T17:37:20.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T17:37:20.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T17:37:20.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T17:37:20.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.num_levels: 7 2023-07-14T17:37:20.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T17:37:20.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T17:37:20.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T17:37:20.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T17:37:20.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T17:37:20.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T17:37:20.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T17:37:20.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T17:37:20.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T17:37:20.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T17:37:20.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T17:37:20.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T17:37:20.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T17:37:20.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T17:37:20.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T17:37:20.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T17:37:20.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T17:37:20.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T17:37:20.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T17:37:20.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T17:37:20.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T17:37:20.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T17:37:20.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T17:37:20.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T17:37:20.420 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T17:37:20.420 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T17:37:20.420 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T17:37:20.420 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T17:37:20.420 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T17:37:20.420 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T17:37:20.420 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T17:37:20.421 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T17:37:20.421 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T17:37:20.421 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T17:37:20.421 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T17:37:20.421 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T17:37:20.421 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T17:37:20.421 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T17:37:20.421 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T17:37:20.422 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T17:37:20.422 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T17:37:20.422 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T17:37:20.422 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T17:37:20.422 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T17:37:20.422 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T17:37:20.422 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T17:37:20.422 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T17:37:20.423 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T17:37:20.423 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T17:37:20.423 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T17:37:20.423 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T17:37:20.423 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T17:37:20.423 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T17:37:20.423 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.table_properties_collectors: 2023-07-14T17:37:20.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.inplace_update_support: 0 2023-07-14T17:37:20.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T17:37:20.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T17:37:20.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T17:37:20.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T17:37:20.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.bloom_locality: 0 2023-07-14T17:37:20.424 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.max_successive_merges: 0 2023-07-14T17:37:20.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T17:37:20.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T17:37:20.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T17:37:20.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T17:37:20.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.ttl: 2592000 2023-07-14T17:37:20.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T17:37:20.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T17:37:20.425 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T17:37:20.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.enable_blob_files: false 2023-07-14T17:37:20.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.min_blob_size: 0 2023-07-14T17:37:20.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T17:37:20.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T17:37:20.426 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T17:37:20.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T17:37:20.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T17:37:20.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T17:37:20.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T17:37:20.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T17:37:20.427 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-14T17:37:20.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-14T17:37:20.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 009caf5d-1067-46d9-9db2-0f45e9f977d3 2023-07-14T17:37:20.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689356240127430, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-14T17:37:20.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-14T17:37:20.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689356240127842, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689356240, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "009caf5d-1067-46d9-9db2-0f45e9f977d3", "db_session_id": "8SISJ24HN2C8VXEW7E19", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-14T17:37:20.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689356240127909, "job": 1, "event": "recovery_finished"} 2023-07-14T17:37:20.428 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-14T17:37:20.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T17:37:20.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564c0aa56000 2023-07-14T17:37:20.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: DB pointer 0x564c0aa42000 2023-07-14T17:37:20.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T17:37:20.429 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T17:37:20.429 INFO:journalctl@ceph.mon.c.smithi178.stdout: ** DB Stats ** 2023-07-14T17:37:20.430 INFO:journalctl@ceph.mon.c.smithi178.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T17:37:20.430 INFO:journalctl@ceph.mon.c.smithi178.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T17:37:20.430 INFO:journalctl@ceph.mon.c.smithi178.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T17:37:20.430 INFO:journalctl@ceph.mon.c.smithi178.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T17:37:20.430 INFO:journalctl@ceph.mon.c.smithi178.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T17:37:20.430 INFO:journalctl@ceph.mon.c.smithi178.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T17:37:20.430 INFO:journalctl@ceph.mon.c.smithi178.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T17:37:20.430 INFO:journalctl@ceph.mon.c.smithi178.stdout: 2023-07-14T17:37:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout: ** Compaction Stats [default] ** 2023-07-14T17:37:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T17:37:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T17:37:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T17:37:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T17:37:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T17:37:20.431 INFO:journalctl@ceph.mon.c.smithi178.stdout: 2023-07-14T17:37:20.432 INFO:journalctl@ceph.mon.c.smithi178.stdout: ** Compaction Stats [default] ** 2023-07-14T17:37:20.432 INFO:journalctl@ceph.mon.c.smithi178.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T17:37:20.432 INFO:journalctl@ceph.mon.c.smithi178.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T17:37:20.433 INFO:journalctl@ceph.mon.c.smithi178.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T17:37:20.433 INFO:journalctl@ceph.mon.c.smithi178.stdout: 2023-07-14T17:37:20.433 INFO:journalctl@ceph.mon.c.smithi178.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T17:37:20.433 INFO:journalctl@ceph.mon.c.smithi178.stdout: 2023-07-14T17:37:20.433 INFO:journalctl@ceph.mon.c.smithi178.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T17:37:20.433 INFO:journalctl@ceph.mon.c.smithi178.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T17:37:20.433 INFO:journalctl@ceph.mon.c.smithi178.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T17:37:20.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T17:37:20.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T17:37:20.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T17:37:20.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T17:37:20.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T17:37:20.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T17:37:20.434 INFO:journalctl@ceph.mon.c.smithi178.stdout: Block cache BinnedLRUCache@0x564c0a9bd1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-14T17:37:20.435 INFO:journalctl@ceph.mon.c.smithi178.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-14T17:37:20.435 INFO:journalctl@ceph.mon.c.smithi178.stdout: 2023-07-14T17:37:20.435 INFO:journalctl@ceph.mon.c.smithi178.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T17:37:20.435 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-07-14T17:37:20.435 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: using public_addr v2:172.21.15.178:0/0 -> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-07-14T17:37:20.435 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: starting mon.c rank -1 at public addrs [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] at bind addrs [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:37:20.435 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: mon.c@-1(???) e0 preinit fsid b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:37:20.435 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: mon.c@-1(synchronizing).mds e1 new map 2023-07-14T17:37:20.436 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: mon.c@-1(synchronizing).mds e1 print_map 2023-07-14T17:37:20.436 INFO:journalctl@ceph.mon.c.smithi178.stdout: e1 2023-07-14T17:37:20.436 INFO:journalctl@ceph.mon.c.smithi178.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T17:37:20.436 INFO:journalctl@ceph.mon.c.smithi178.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T17:37:20.436 INFO:journalctl@ceph.mon.c.smithi178.stdout: legacy client fscid: -1 2023-07-14T17:37:20.437 INFO:journalctl@ceph.mon.c.smithi178.stdout: 2023-07-14T17:37:20.437 INFO:journalctl@ceph.mon.c.smithi178.stdout: No filesystems configured 2023-07-14T17:37:20.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-14T17:37:20.437 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-14T17:37:20.438 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-14T17:37:20.438 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-14T17:37:20.438 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-14T17:37:20.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-14T17:37:20.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T17:37:20.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T17:37:20.439 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T17:37:20.440 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T17:37:20.440 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:37:20.440 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:20.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3442476058' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T17:37:20.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:20.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:20.441 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:20.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:20.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi178", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:37:20.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:20.442 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:37:20.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: Updating smithi178:/etc/ceph/ceph.conf 2023-07-14T17:37:20.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: Updating smithi178:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T17:37:20.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:20.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: Updating smithi178:/etc/ceph/ceph.client.admin.keyring 2023-07-14T17:37:20.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: Updating smithi178:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T17:37:20.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3442476058' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T17:37:20.443 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T17:37:20.444 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:20.444 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:20.444 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:20.444 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:20.444 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:20.445 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:37:20.445 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:20.445 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:37:20.445 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:20.445 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T17:37:20.445 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:20.445 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi088:172.21.15.88=a;smithi107:172.21.15.107=b;smithi178:172.21.15.178=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:37:20.446 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: Saving service mon spec with placement smithi088:172.21.15.88=a;smithi107:172.21.15.107=b;smithi178:172.21.15.178=c;count:3 2023-07-14T17:37:20.446 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: Deploying daemon mon.c on smithi178 2023-07-14T17:37:20.446 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:20.446 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-14T17:37:20.446 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: expand_channel_meta expand map: {default=false} 2023-07-14T17:37:20.446 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: expand_channel_meta from 'false' to 'false' 2023-07-14T17:37:20.446 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: expand_channel_meta expanded map: {default=false} 2023-07-14T17:37:20.447 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: expand_channel_meta expand map: {default=info} 2023-07-14T17:37:20.447 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: expand_channel_meta from 'info' to 'info' 2023-07-14T17:37:20.447 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: expand_channel_meta expanded map: {default=info} 2023-07-14T17:37:20.447 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: expand_channel_meta expand map: {default=daemon} 2023-07-14T17:37:20.447 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-14T17:37:20.447 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: expand_channel_meta expanded map: {default=daemon} 2023-07-14T17:37:20.447 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: expand_channel_meta expand map: {default=debug} 2023-07-14T17:37:20.448 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: expand_channel_meta from 'debug' to 'debug' 2023-07-14T17:37:20.448 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: expand_channel_meta expanded map: {default=debug} 2023-07-14T17:37:20.448 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 ceph-mon[129368]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-07-14T17:37:20.448 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:20 smithi178 systemd[1]: Started Ceph mon.c for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:37:21.450 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T17:37:21.451 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T17:37:22.014 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T17:37:25.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:25.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T17:37:25.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:25.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: mon.a calling monitor election 2023-07-14T17:37:25.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:25.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:25.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:25.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: mon.c calling monitor election 2023-07-14T17:37:25.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:25.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:25.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:25.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:25.671 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T17:37:25.671 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: monmap e2: 2 mons at {a=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} 2023-07-14T17:37:25.671 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: fsmap 2023-07-14T17:37:25.671 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T17:37:25.671 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: mgrmap e13: x(active, since 41s) 2023-07-14T17:37:25.671 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: overall HEALTH_OK 2023-07-14T17:37:25.671 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:25.671 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:25.672 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:25.672 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T17:37:25.672 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:25.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:25.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T17:37:25.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:25.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: mon.a calling monitor election 2023-07-14T17:37:25.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:25.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:25.697 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:25.697 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: mon.c calling monitor election 2023-07-14T17:37:25.697 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:25.697 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:25.697 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:25.697 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:25.697 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T17:37:25.697 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: monmap e2: 2 mons at {a=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} 2023-07-14T17:37:25.698 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: fsmap 2023-07-14T17:37:25.698 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T17:37:25.698 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: mgrmap e13: x(active, since 41s) 2023-07-14T17:37:25.698 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: overall HEALTH_OK 2023-07-14T17:37:25.698 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:25.698 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:25.698 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:25.699 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T17:37:25.699 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:26.235 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.248+0000 7f5b68ac6700 1 -- 172.21.15.178:0/3386751256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b4c0035c0 msgr2=0x7f5b4c005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:26.235 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.248+0000 7f5b68ac6700 1 --2- 172.21.15.178:0/3386751256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b4c0035c0 0x7f5b4c005a40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5b64100cf0 tx=0x7f5b5800fe60 comp rx=0 tx=0).stop 2023-07-14T17:37:26.235 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.248+0000 7f5b68ac6700 1 -- 172.21.15.178:0/3386751256 shutdown_connections 2023-07-14T17:37:26.235 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.248+0000 7f5b68ac6700 1 --2- 172.21.15.178:0/3386751256 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b4c0035c0 0x7f5b4c005a40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:26.236 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.248+0000 7f5b68ac6700 1 -- 172.21.15.178:0/3386751256 >> 172.21.15.178:0/3386751256 conn(0x7f5b640fb340 msgr2=0x7f5b640fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:26.236 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.248+0000 7f5b68ac6700 1 -- 172.21.15.178:0/3386751256 shutdown_connections 2023-07-14T17:37:26.236 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.248+0000 7f5b68ac6700 1 -- 172.21.15.178:0/3386751256 wait complete. 2023-07-14T17:37:26.237 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.250+0000 7f5b68ac6700 1 Processor -- start 2023-07-14T17:37:26.237 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.250+0000 7f5b68ac6700 1 -- start start 2023-07-14T17:37:26.238 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.250+0000 7f5b68ac6700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b4c0035c0 0x7f5b64072940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:26.238 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.250+0000 7f5b68ac6700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b64072e80 0x7f5b640732f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:26.238 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.250+0000 7f5b68ac6700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b640748c0 con 0x7f5b64072e80 2023-07-14T17:37:26.238 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.250+0000 7f5b68ac6700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b641a0e60 con 0x7f5b4c0035c0 2023-07-14T17:37:26.238 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.250+0000 7f5b6259c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b4c0035c0 0x7f5b64072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:26.238 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.250+0000 7f5b6259c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b4c0035c0 0x7f5b64072940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:54580/0 (socket says 172.21.15.178:54580) 2023-07-14T17:37:26.238 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.250+0000 7f5b6259c700 1 -- 172.21.15.178:0/568462281 learned_addr learned my addr 172.21.15.178:0/568462281 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-07-14T17:37:26.239 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.250+0000 7f5b6259c700 1 -- 172.21.15.178:0/568462281 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b4c0035c0 msgr2=0x7f5b64072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-07-14T17:37:26.239 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.250+0000 7f5b6259c700 1 -- 172.21.15.178:0/568462281 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b4c0035c0 msgr2=0x7f5b64072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T17:37:26.239 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.250+0000 7f5b6259c700 1 --2- 172.21.15.178:0/568462281 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b4c0035c0 0x7f5b64072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T17:37:26.239 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.250+0000 7f5b6259c700 1 --2- 172.21.15.178:0/568462281 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b4c0035c0 0x7f5b64072940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T17:37:26.239 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.250+0000 7f5b61d9b700 1 --2- 172.21.15.178:0/568462281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b64072e80 0x7f5b640732f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:26.239 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.251+0000 7f5b61d9b700 1 -- 172.21.15.178:0/568462281 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b4c0035c0 msgr2=0x7f5b64072940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T17:37:26.239 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.251+0000 7f5b61d9b700 1 --2- 172.21.15.178:0/568462281 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b4c0035c0 0x7f5b64072940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:26.240 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.251+0000 7f5b61d9b700 1 -- 172.21.15.178:0/568462281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5800f8b0 con 0x7f5b64072e80 2023-07-14T17:37:26.240 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.251+0000 7f5b61d9b700 1 --2- 172.21.15.178:0/568462281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b64072e80 0x7f5b640732f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400b4d0 tx=0x7f5b5400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:26.240 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.251+0000 7f5b537fe700 1 -- 172.21.15.178:0/568462281 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f5b54011630 con 0x7f5b64072e80 2023-07-14T17:37:26.240 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.251+0000 7f5b537fe700 1 -- 172.21.15.178:0/568462281 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5b54011c20 con 0x7f5b64072e80 2023-07-14T17:37:26.240 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.251+0000 7f5b68ac6700 1 -- 172.21.15.178:0/568462281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b641a1060 con 0x7f5b64072e80 2023-07-14T17:37:26.240 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.251+0000 7f5b537fe700 1 -- 172.21.15.178:0/568462281 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f5b54010c60 con 0x7f5b64072e80 2023-07-14T17:37:26.240 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.251+0000 7f5b68ac6700 1 -- 172.21.15.178:0/568462281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b641a1580 con 0x7f5b64072e80 2023-07-14T17:37:26.241 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.252+0000 7f5b537fe700 1 -- 172.21.15.178:0/568462281 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f5b5401a400 con 0x7f5b64072e80 2023-07-14T17:37:26.241 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.253+0000 7f5b68ac6700 1 -- 172.21.15.178:0/568462281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b64061c70 con 0x7f5b64072e80 2023-07-14T17:37:26.241 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.253+0000 7f5b537fe700 1 --2- 172.21.15.178:0/568462281 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5b440385f0 0x7f5b4403aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:26.242 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.253+0000 7f5b537fe700 1 -- 172.21.15.178:0/568462281 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5b5404adc0 con 0x7f5b64072e80 2023-07-14T17:37:26.242 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.253+0000 7f5b6259c700 1 --2- 172.21.15.178:0/568462281 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5b440385f0 0x7f5b4403aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:26.242 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.253+0000 7f5b6259c700 1 --2- 172.21.15.178:0/568462281 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5b440385f0 0x7f5b4403aaa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800f470 tx=0x7f5b5800f3e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:26.244 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.258+0000 7f5b537fe700 1 -- 172.21.15.178:0/568462281 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5b54017680 con 0x7f5b64072e80 2023-07-14T17:37:26.509 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.523+0000 7f5b68ac6700 1 -- 172.21.15.178:0/568462281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b64073e70 con 0x7f5b64072e80 2023-07-14T17:37:26.510 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.524+0000 7f5b537fe700 1 -- 172.21.15.178:0/568462281 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7f5b540172c0 con 0x7f5b64072e80 2023-07-14T17:37:26.510 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-07-14T17:37:26.510 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":2,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","modified":"2023-07-14T17:37:20.148627Z","created":"2023-07-14T17:35:46.715826Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:3300","nonce":0},{"type":"v1","addr":"172.21.15.178:6789","nonce":0}]},"addr":"172.21.15.178:6789/0","public_addr":"172.21.15.178:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-14T17:37:26.512 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.526+0000 7f5b68ac6700 1 -- 172.21.15.178:0/568462281 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5b440385f0 msgr2=0x7f5b4403aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:26.512 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.526+0000 7f5b68ac6700 1 --2- 172.21.15.178:0/568462281 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5b440385f0 0x7f5b4403aaa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800f470 tx=0x7f5b5800f3e0 comp rx=0 tx=0).stop 2023-07-14T17:37:26.512 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.526+0000 7f5b68ac6700 1 -- 172.21.15.178:0/568462281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b64072e80 msgr2=0x7f5b640732f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:26.512 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.526+0000 7f5b68ac6700 1 --2- 172.21.15.178:0/568462281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b64072e80 0x7f5b640732f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400b4d0 tx=0x7f5b5400b890 comp rx=0 tx=0).stop 2023-07-14T17:37:26.512 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.526+0000 7f5b68ac6700 1 -- 172.21.15.178:0/568462281 shutdown_connections 2023-07-14T17:37:26.512 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.526+0000 7f5b68ac6700 1 --2- 172.21.15.178:0/568462281 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b4c0035c0 0x7f5b64072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:26.513 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.526+0000 7f5b68ac6700 1 --2- 172.21.15.178:0/568462281 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5b440385f0 0x7f5b4403aaa0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:26.513 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.526+0000 7f5b68ac6700 1 --2- 172.21.15.178:0/568462281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b64072e80 0x7f5b640732f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:26.513 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.526+0000 7f5b68ac6700 1 -- 172.21.15.178:0/568462281 >> 172.21.15.178:0/568462281 conn(0x7f5b640fb340 msgr2=0x7f5b6418dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:26.513 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.526+0000 7f5b68ac6700 1 -- 172.21.15.178:0/568462281 shutdown_connections 2023-07-14T17:37:26.513 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:26.526+0000 7f5b68ac6700 1 -- 172.21.15.178:0/568462281 wait complete. 2023-07-14T17:37:26.514 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 2 2023-07-14T17:37:26.537 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:26 smithi178 ceph-mon[129368]: Deploying daemon mon.b on smithi107 2023-07-14T17:37:26.537 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:26 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:26.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:26 smithi088 ceph-mon[127465]: Deploying daemon mon.b on smithi107 2023-07-14T17:37:26.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:26 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:27.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:27 smithi178 ceph-mon[129368]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:27.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:27 smithi178 ceph-mon[129368]: from='client.? 172.21.15.178:0/568462281' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T17:37:27.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:27 smithi088 ceph-mon[127465]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:27.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:27 smithi088 ceph-mon[127465]: from='client.? 172.21.15.178:0/568462281' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T17:37:27.862 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T17:37:27.862 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T17:37:28.143 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:27 smithi107 systemd[1]: Starting Ceph mon.b for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:37:28.424 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T17:37:28.437 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 podman[128959]: 2023-07-14T17:37:28.437 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T17:37:28.437 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T17:37:28.437 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 bash[128959]: db1219f65df0a206a6828c546df1f174379edc3c042cb602ab095fda18f9daed 2023-07-14T17:37:28.438 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: pidfile_write: ignore empty --pid-file 2023-07-14T17:37:28.438 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: load: jerasure load: lrc 2023-07-14T17:37:28.438 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: RocksDB version: 7.9.2 2023-07-14T17:37:28.438 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Git sha 0 2023-07-14T17:37:28.438 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T17:37:28.438 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: DB SUMMARY 2023-07-14T17:37:28.438 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: DB Session ID: DOO6YQ7USMHL6OKATLWK 2023-07-14T17:37:28.439 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: CURRENT file: CURRENT 2023-07-14T17:37:28.439 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T17:37:28.439 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-14T17:37:28.439 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-07-14T17:37:28.439 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-07-14T17:37:28.439 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.error_if_exists: 0 2023-07-14T17:37:28.439 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.create_if_missing: 0 2023-07-14T17:37:28.440 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.paranoid_checks: 1 2023-07-14T17:37:28.440 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T17:37:28.440 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T17:37:28.440 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T17:37:28.440 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.env: 0x564569632c60 2023-07-14T17:37:28.440 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T17:37:28.440 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.info_log: 0x56456b4f72a0 2023-07-14T17:37:28.440 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T17:37:28.441 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.statistics: (nil) 2023-07-14T17:37:28.441 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.use_fsync: 0 2023-07-14T17:37:28.441 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_log_file_size: 0 2023-07-14T17:37:28.441 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T17:37:28.441 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T17:37:28.441 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T17:37:28.441 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T17:37:28.442 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.allow_fallocate: 1 2023-07-14T17:37:28.442 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T17:37:28.442 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T17:37:28.442 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.use_direct_reads: 0 2023-07-14T17:37:28.442 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T17:37:28.442 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T17:37:28.442 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.db_log_dir: 2023-07-14T17:37:28.442 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.wal_dir: 2023-07-14T17:37:28.443 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T17:37:28.443 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T17:37:28.443 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T17:37:28.443 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T17:37:28.443 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T17:37:28.443 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T17:37:28.443 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T17:37:28.443 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T17:37:28.444 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.write_buffer_manager: 0x56456b4183c0 2023-07-14T17:37:28.444 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T17:37:28.444 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T17:37:28.444 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T17:37:28.444 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T17:37:28.444 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T17:37:28.444 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T17:37:28.445 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T17:37:28.445 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T17:37:28.445 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.unordered_write: 0 2023-07-14T17:37:28.445 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T17:37:28.445 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T17:37:28.445 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T17:37:28.445 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T17:37:28.445 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.row_cache: None 2023-07-14T17:37:28.446 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.wal_filter: None 2023-07-14T17:37:28.446 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T17:37:28.446 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T17:37:28.446 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.two_write_queues: 0 2023-07-14T17:37:28.446 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T17:37:28.446 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.wal_compression: 0 2023-07-14T17:37:28.446 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.atomic_flush: 0 2023-07-14T17:37:28.446 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T17:37:28.447 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T17:37:28.447 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T17:37:28.447 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.log_readahead_size: 0 2023-07-14T17:37:28.447 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T17:37:28.447 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T17:37:28.447 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T17:37:28.447 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T17:37:28.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T17:37:28.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T17:37:28.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T17:37:28.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_background_jobs: 2 2023-07-14T17:37:28.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_background_compactions: -1 2023-07-14T17:37:28.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_subcompactions: 1 2023-07-14T17:37:28.450 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T17:37:28.450 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T17:37:28.450 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T17:37:28.450 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T17:37:28.450 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T17:37:28.450 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T17:37:28.450 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T17:37:28.451 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T17:37:28.451 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_open_files: -1 2023-07-14T17:37:28.451 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T17:37:28.451 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T17:37:28.451 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T17:37:28.451 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T17:37:28.451 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_background_flushes: -1 2023-07-14T17:37:28.452 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Compression algorithms supported: 2023-07-14T17:37:28.452 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T17:37:28.452 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: kZSTD supported: 0 2023-07-14T17:37:28.452 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: kXpressCompression supported: 0 2023-07-14T17:37:28.452 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T17:37:28.452 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: kZlibCompression supported: 1 2023-07-14T17:37:28.452 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: kSnappyCompression supported: 1 2023-07-14T17:37:28.453 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: kLZ4Compression supported: 1 2023-07-14T17:37:28.453 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: kBZip2Compression supported: 0 2023-07-14T17:37:28.453 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T17:37:28.453 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T17:37:28.453 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-07-14T17:37:28.453 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T17:37:28.453 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T17:37:28.453 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.merge_operator: 2023-07-14T17:37:28.454 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compaction_filter: None 2023-07-14T17:37:28.454 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T17:37:28.454 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T17:37:28.454 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T17:37:28.454 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T17:37:28.454 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56456b4f73a0) 2023-07-14T17:37:28.454 INFO:journalctl@ceph.mon.b.smithi107.stdout: cache_index_and_filter_blocks: 1 2023-07-14T17:37:28.454 INFO:journalctl@ceph.mon.b.smithi107.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T17:37:28.455 INFO:journalctl@ceph.mon.b.smithi107.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T17:37:28.455 INFO:journalctl@ceph.mon.b.smithi107.stdout: pin_top_level_index_and_filter: 1 2023-07-14T17:37:28.455 INFO:journalctl@ceph.mon.b.smithi107.stdout: index_type: 0 2023-07-14T17:37:28.455 INFO:journalctl@ceph.mon.b.smithi107.stdout: data_block_index_type: 0 2023-07-14T17:37:28.455 INFO:journalctl@ceph.mon.b.smithi107.stdout: index_shortening: 1 2023-07-14T17:37:28.455 INFO:journalctl@ceph.mon.b.smithi107.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T17:37:28.455 INFO:journalctl@ceph.mon.b.smithi107.stdout: checksum: 4 2023-07-14T17:37:28.455 INFO:journalctl@ceph.mon.b.smithi107.stdout: no_block_cache: 0 2023-07-14T17:37:28.456 INFO:journalctl@ceph.mon.b.smithi107.stdout: block_cache: 0x56456b4b71f0 2023-07-14T17:37:28.456 INFO:journalctl@ceph.mon.b.smithi107.stdout: block_cache_name: BinnedLRUCache 2023-07-14T17:37:28.456 INFO:journalctl@ceph.mon.b.smithi107.stdout: block_cache_options: 2023-07-14T17:37:28.456 INFO:journalctl@ceph.mon.b.smithi107.stdout: capacity : 536870912 2023-07-14T17:37:28.456 INFO:journalctl@ceph.mon.b.smithi107.stdout: num_shard_bits : 4 2023-07-14T17:37:28.456 INFO:journalctl@ceph.mon.b.smithi107.stdout: strict_capacity_limit : 0 2023-07-14T17:37:28.456 INFO:journalctl@ceph.mon.b.smithi107.stdout: high_pri_pool_ratio: 0.000 2023-07-14T17:37:28.457 INFO:journalctl@ceph.mon.b.smithi107.stdout: block_cache_compressed: (nil) 2023-07-14T17:37:28.457 INFO:journalctl@ceph.mon.b.smithi107.stdout: persistent_cache: (nil) 2023-07-14T17:37:28.457 INFO:journalctl@ceph.mon.b.smithi107.stdout: block_size: 4096 2023-07-14T17:37:28.457 INFO:journalctl@ceph.mon.b.smithi107.stdout: block_size_deviation: 10 2023-07-14T17:37:28.457 INFO:journalctl@ceph.mon.b.smithi107.stdout: block_restart_interval: 16 2023-07-14T17:37:28.457 INFO:journalctl@ceph.mon.b.smithi107.stdout: index_block_restart_interval: 1 2023-07-14T17:37:28.457 INFO:journalctl@ceph.mon.b.smithi107.stdout: metadata_block_size: 4096 2023-07-14T17:37:28.457 INFO:journalctl@ceph.mon.b.smithi107.stdout: partition_filters: 0 2023-07-14T17:37:28.458 INFO:journalctl@ceph.mon.b.smithi107.stdout: use_delta_encoding: 1 2023-07-14T17:37:28.458 INFO:journalctl@ceph.mon.b.smithi107.stdout: filter_policy: bloomfilter 2023-07-14T17:37:28.458 INFO:journalctl@ceph.mon.b.smithi107.stdout: whole_key_filtering: 1 2023-07-14T17:37:28.458 INFO:journalctl@ceph.mon.b.smithi107.stdout: verify_compression: 0 2023-07-14T17:37:28.458 INFO:journalctl@ceph.mon.b.smithi107.stdout: read_amp_bytes_per_bit: 0 2023-07-14T17:37:28.458 INFO:journalctl@ceph.mon.b.smithi107.stdout: format_version: 5 2023-07-14T17:37:28.458 INFO:journalctl@ceph.mon.b.smithi107.stdout: enable_index_compression: 1 2023-07-14T17:37:28.458 INFO:journalctl@ceph.mon.b.smithi107.stdout: block_align: 0 2023-07-14T17:37:28.459 INFO:journalctl@ceph.mon.b.smithi107.stdout: max_auto_readahead_size: 262144 2023-07-14T17:37:28.459 INFO:journalctl@ceph.mon.b.smithi107.stdout: prepopulate_block_cache: 0 2023-07-14T17:37:28.459 INFO:journalctl@ceph.mon.b.smithi107.stdout: initial_auto_readahead_size: 8192 2023-07-14T17:37:28.459 INFO:journalctl@ceph.mon.b.smithi107.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T17:37:28.459 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T17:37:28.459 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T17:37:28.459 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compression: NoCompression 2023-07-14T17:37:28.459 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T17:37:28.460 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T17:37:28.460 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T17:37:28.460 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.num_levels: 7 2023-07-14T17:37:28.460 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T17:37:28.460 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T17:37:28.460 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T17:37:28.460 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T17:37:28.461 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T17:37:28.461 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T17:37:28.461 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T17:37:28.461 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T17:37:28.461 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T17:37:28.461 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T17:37:28.461 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T17:37:28.461 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T17:37:28.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T17:37:28.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T17:37:28.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T17:37:28.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T17:37:28.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T17:37:28.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T17:37:28.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T17:37:28.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T17:37:28.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T17:37:28.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T17:37:28.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T17:37:28.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T17:37:28.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T17:37:28.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T17:37:28.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T17:37:28.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T17:37:28.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T17:37:28.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T17:37:28.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T17:37:28.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T17:37:28.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T17:37:28.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T17:37:28.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T17:37:28.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T17:37:28.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T17:37:28.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T17:37:28.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T17:37:28.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T17:37:28.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T17:37:28.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T17:37:28.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T17:37:28.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T17:37:28.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T17:37:28.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T17:37:28.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T17:37:28.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T17:37:28.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T17:37:28.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T17:37:28.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T17:37:28.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T17:37:28.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T17:37:28.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.table_properties_collectors: 2023-07-14T17:37:28.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.inplace_update_support: 0 2023-07-14T17:37:28.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T17:37:28.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T17:37:28.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T17:37:28.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T17:37:28.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.bloom_locality: 0 2023-07-14T17:37:28.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.max_successive_merges: 0 2023-07-14T17:37:28.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T17:37:28.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T17:37:28.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T17:37:28.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T17:37:28.469 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.ttl: 2592000 2023-07-14T17:37:28.469 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T17:37:28.469 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T17:37:28.469 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T17:37:28.469 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.enable_blob_files: false 2023-07-14T17:37:28.469 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.min_blob_size: 0 2023-07-14T17:37:28.469 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T17:37:28.470 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T17:37:28.470 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T17:37:28.470 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T17:37:28.470 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T17:37:28.470 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T17:37:28.470 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T17:37:28.470 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T17:37:28.470 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-14T17:37:28.471 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-14T17:37:28.471 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6201b420-bfc8-4d38-8776-89f53c063d6e 2023-07-14T17:37:28.471 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689356248371119, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-14T17:37:28.471 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-14T17:37:28.471 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689356248371675, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689356248, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6201b420-bfc8-4d38-8776-89f53c063d6e", "db_session_id": "DOO6YQ7USMHL6OKATLWK", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-14T17:37:28.471 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689356248371765, "job": 1, "event": "recovery_finished"} 2023-07-14T17:37:28.471 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-14T17:37:28.472 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T17:37:28.472 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56456b550000 2023-07-14T17:37:28.472 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: DB pointer 0x56456b53c000 2023-07-14T17:37:28.472 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T17:37:28.472 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T17:37:28.472 INFO:journalctl@ceph.mon.b.smithi107.stdout: ** DB Stats ** 2023-07-14T17:37:28.472 INFO:journalctl@ceph.mon.b.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T17:37:28.473 INFO:journalctl@ceph.mon.b.smithi107.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T17:37:28.473 INFO:journalctl@ceph.mon.b.smithi107.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T17:37:28.473 INFO:journalctl@ceph.mon.b.smithi107.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T17:37:28.473 INFO:journalctl@ceph.mon.b.smithi107.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T17:37:28.473 INFO:journalctl@ceph.mon.b.smithi107.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T17:37:28.473 INFO:journalctl@ceph.mon.b.smithi107.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T17:37:28.473 INFO:journalctl@ceph.mon.b.smithi107.stdout: 2023-07-14T17:37:28.473 INFO:journalctl@ceph.mon.b.smithi107.stdout: ** Compaction Stats [default] ** 2023-07-14T17:37:28.474 INFO:journalctl@ceph.mon.b.smithi107.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T17:37:28.474 INFO:journalctl@ceph.mon.b.smithi107.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T17:37:28.474 INFO:journalctl@ceph.mon.b.smithi107.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T17:37:28.474 INFO:journalctl@ceph.mon.b.smithi107.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T17:37:28.474 INFO:journalctl@ceph.mon.b.smithi107.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T17:37:28.474 INFO:journalctl@ceph.mon.b.smithi107.stdout: 2023-07-14T17:37:28.474 INFO:journalctl@ceph.mon.b.smithi107.stdout: ** Compaction Stats [default] ** 2023-07-14T17:37:28.475 INFO:journalctl@ceph.mon.b.smithi107.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T17:37:28.475 INFO:journalctl@ceph.mon.b.smithi107.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T17:37:28.475 INFO:journalctl@ceph.mon.b.smithi107.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T17:37:28.475 INFO:journalctl@ceph.mon.b.smithi107.stdout: 2023-07-14T17:37:28.475 INFO:journalctl@ceph.mon.b.smithi107.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T17:37:28.475 INFO:journalctl@ceph.mon.b.smithi107.stdout: 2023-07-14T17:37:28.475 INFO:journalctl@ceph.mon.b.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T17:37:28.475 INFO:journalctl@ceph.mon.b.smithi107.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T17:37:28.476 INFO:journalctl@ceph.mon.b.smithi107.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T17:37:28.476 INFO:journalctl@ceph.mon.b.smithi107.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T17:37:28.476 INFO:journalctl@ceph.mon.b.smithi107.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T17:37:28.476 INFO:journalctl@ceph.mon.b.smithi107.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T17:37:28.476 INFO:journalctl@ceph.mon.b.smithi107.stdout: Cumulative compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T17:37:28.476 INFO:journalctl@ceph.mon.b.smithi107.stdout: Interval compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T17:37:28.476 INFO:journalctl@ceph.mon.b.smithi107.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T17:37:28.477 INFO:journalctl@ceph.mon.b.smithi107.stdout: Block cache BinnedLRUCache@0x56456b4b71f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-07-14T17:37:28.477 INFO:journalctl@ceph.mon.b.smithi107.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-14T17:37:28.477 INFO:journalctl@ceph.mon.b.smithi107.stdout: 2023-07-14T17:37:28.477 INFO:journalctl@ceph.mon.b.smithi107.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T17:37:28.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-07-14T17:37:28.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: using public_addr v2:172.21.15.107:0/0 -> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] 2023-07-14T17:37:28.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: starting mon.b rank -1 at public addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] at bind addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:37:28.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.b@-1(???) e0 preinit fsid b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:37:28.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.b@-1(synchronizing).mds e1 new map 2023-07-14T17:37:28.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.b@-1(synchronizing).mds e1 print_map 2023-07-14T17:37:28.478 INFO:journalctl@ceph.mon.b.smithi107.stdout: e1 2023-07-14T17:37:28.478 INFO:journalctl@ceph.mon.b.smithi107.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T17:37:28.478 INFO:journalctl@ceph.mon.b.smithi107.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T17:37:28.478 INFO:journalctl@ceph.mon.b.smithi107.stdout: legacy client fscid: -1 2023-07-14T17:37:28.479 INFO:journalctl@ceph.mon.b.smithi107.stdout: 2023-07-14T17:37:28.479 INFO:journalctl@ceph.mon.b.smithi107.stdout: No filesystems configured 2023-07-14T17:37:28.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-14T17:37:28.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-14T17:37:28.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-14T17:37:28.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-14T17:37:28.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-14T17:37:28.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-14T17:37:28.480 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T17:37:28.480 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T17:37:28.480 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T17:37:28.480 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T17:37:28.480 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:37:28.480 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:28.480 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3442476058' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T17:37:28.481 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:28.481 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:28.481 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:28.481 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:28.481 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi178", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:37:28.481 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:28.481 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:37:28.481 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: Updating smithi178:/etc/ceph/ceph.conf 2023-07-14T17:37:28.482 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: Updating smithi178:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T17:37:28.482 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:28.482 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: Updating smithi178:/etc/ceph/ceph.client.admin.keyring 2023-07-14T17:37:28.482 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: Updating smithi178:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T17:37:28.482 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3442476058' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T17:37:28.482 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T17:37:28.482 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:28.483 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:28.483 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:28.483 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:28.483 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:28.483 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:37:28.483 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:28.483 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:37:28.483 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:28.484 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T17:37:28.484 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:28.484 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi088:172.21.15.88=a;smithi107:172.21.15.107=b;smithi178:172.21.15.178=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:37:28.484 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: Saving service mon spec with placement smithi088:172.21.15.88=a;smithi107:172.21.15.107=b;smithi178:172.21.15.178=c;count:3 2023-07-14T17:37:28.484 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: Deploying daemon mon.c on smithi178 2023-07-14T17:37:28.484 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:28.484 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:28.485 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T17:37:28.485 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:28.485 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.a calling monitor election 2023-07-14T17:37:28.485 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:28.485 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:28.485 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:28.485 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.c calling monitor election 2023-07-14T17:37:28.485 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:28.486 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:28.486 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:28.486 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:28.486 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T17:37:28.486 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: monmap e2: 2 mons at {a=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} 2023-07-14T17:37:28.486 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: fsmap 2023-07-14T17:37:28.486 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T17:37:28.487 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mgrmap e13: x(active, since 41s) 2023-07-14T17:37:28.487 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: overall HEALTH_OK 2023-07-14T17:37:28.487 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:28.487 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:28.487 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:28.487 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T17:37:28.487 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:28.487 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: Deploying daemon mon.b on smithi107 2023-07-14T17:37:28.488 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:28.488 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:28.488 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: from='client.? 172.21.15.178:0/568462281' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T17:37:28.488 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-14T17:37:28.488 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 ceph-mon[128999]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-07-14T17:37:28.488 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:28 smithi107 systemd[1]: Started Ceph mon.b for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:37:33.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T17:37:33.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: mon.a calling monitor election 2023-07-14T17:37:33.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: mon.c calling monitor election 2023-07-14T17:37:33.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:33.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:33.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:33.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:33.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:33.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:33.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:33.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:33.920 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:33.920 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T17:37:33.920 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: monmap e3: 3 mons at {a=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],b=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} 2023-07-14T17:37:33.920 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: fsmap 2023-07-14T17:37:33.920 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T17:37:33.920 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: mgrmap e13: x(active, since 49s) 2023-07-14T17:37:33.920 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-14T17:37:33.921 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-14T17:37:33.921 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-14T17:37:33.921 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: mon.b (rank 2) addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] is down (out of quorum) 2023-07-14T17:37:33.921 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:33.921 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:33 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:33.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T17:37:33.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: mon.a calling monitor election 2023-07-14T17:37:33.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: mon.c calling monitor election 2023-07-14T17:37:33.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:33.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:33.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:33.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:33.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:33.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:33.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:33.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:33.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:33.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T17:37:33.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: monmap e3: 3 mons at {a=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],b=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} 2023-07-14T17:37:33.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: fsmap 2023-07-14T17:37:33.950 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T17:37:33.950 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: mgrmap e13: x(active, since 49s) 2023-07-14T17:37:33.950 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-14T17:37:33.950 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-14T17:37:33.950 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-14T17:37:33.950 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: mon.b (rank 2) addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] is down (out of quorum) 2023-07-14T17:37:33.950 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:33.951 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:33 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:34.102 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.116+0000 7f4d8cb76700 1 -- 172.21.15.178:0/704509172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d700057d0 msgr2=0x7f4d70007c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:34.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.116+0000 7f4d8cb76700 1 --2- 172.21.15.178:0/704509172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d700057d0 0x7f4d70007c50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4d880fe670 tx=0x7f4d7c00fe10 comp rx=0 tx=0).stop 2023-07-14T17:37:34.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.116+0000 7f4d8cb76700 1 -- 172.21.15.178:0/704509172 shutdown_connections 2023-07-14T17:37:34.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.116+0000 7f4d8cb76700 1 --2- 172.21.15.178:0/704509172 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d700057d0 0x7f4d70007c50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:34.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.116+0000 7f4d8cb76700 1 -- 172.21.15.178:0/704509172 >> 172.21.15.178:0/704509172 conn(0x7f4d880fa260 msgr2=0x7f4d880fc670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:34.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.116+0000 7f4d8cb76700 1 -- 172.21.15.178:0/704509172 shutdown_connections 2023-07-14T17:37:34.103 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.116+0000 7f4d8cb76700 1 -- 172.21.15.178:0/704509172 wait complete. 2023-07-14T17:37:34.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.117+0000 7f4d8cb76700 1 Processor -- start 2023-07-14T17:37:34.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.117+0000 7f4d8cb76700 1 -- start start 2023-07-14T17:37:34.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.117+0000 7f4d8cb76700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d700057d0 0x7f4d88198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:34.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.117+0000 7f4d8cb76700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d88198e40 0x7f4d88192980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:34.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.117+0000 7f4d8cb76700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d88192ec0 0x7f4d88193360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:34.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.117+0000 7f4d8cb76700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d88199450 con 0x7f4d88198e40 2023-07-14T17:37:34.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.117+0000 7f4d8cb76700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8819f720 con 0x7f4d88192ec0 2023-07-14T17:37:34.106 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.117+0000 7f4d8cb76700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8819f860 con 0x7f4d700057d0 2023-07-14T17:37:34.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d85d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d88198e40 0x7f4d88192980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:34.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d85d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d88198e40 0x7f4d88192980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.178:51442/0 (socket says 172.21.15.178:51442) 2023-07-14T17:37:34.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d8659c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d700057d0 0x7f4d88198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:34.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d85d9b700 1 -- 172.21.15.178:0/162235727 learned_addr learned my addr 172.21.15.178:0/162235727 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-07-14T17:37:34.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d86d9d700 1 --2- 172.21.15.178:0/162235727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d88192ec0 0x7f4d88193360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:34.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d8659c700 1 -- 172.21.15.178:0/162235727 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d700057d0 msgr2=0x7f4d88198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-07-14T17:37:34.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d8659c700 1 -- 172.21.15.178:0/162235727 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d700057d0 msgr2=0x7f4d88198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T17:37:34.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d8659c700 1 --2- 172.21.15.178:0/162235727 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d700057d0 0x7f4d88198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T17:37:34.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d8659c700 1 --2- 172.21.15.178:0/162235727 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d700057d0 0x7f4d88198900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T17:37:34.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d86d9d700 1 -- 172.21.15.178:0/162235727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d88192ec0 msgr2=0x7f4d88193360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-07-14T17:37:34.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d86d9d700 1 -- 172.21.15.178:0/162235727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d88192ec0 msgr2=0x7f4d88193360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T17:37:34.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d86d9d700 1 --2- 172.21.15.178:0/162235727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d88192ec0 0x7f4d88193360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T17:37:34.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d86d9d700 1 --2- 172.21.15.178:0/162235727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d88192ec0 0x7f4d88193360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T17:37:34.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d85d9b700 1 -- 172.21.15.178:0/162235727 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d700057d0 msgr2=0x7f4d88198900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T17:37:34.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d85d9b700 1 --2- 172.21.15.178:0/162235727 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d700057d0 0x7f4d88198900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:34.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d85d9b700 1 -- 172.21.15.178:0/162235727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d88192ec0 msgr2=0x7f4d88193360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T17:37:34.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d85d9b700 1 --2- 172.21.15.178:0/162235727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d88192ec0 0x7f4d88193360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:34.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d85d9b700 1 -- 172.21.15.178:0/162235727 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7c00f860 con 0x7f4d88198e40 2023-07-14T17:37:34.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d85d9b700 1 --2- 172.21.15.178:0/162235727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d88198e40 0x7f4d88192980 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4d7800b780 tx=0x7f4d7800bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:34.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d777fe700 1 -- 172.21.15.178:0/162235727 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d78004230 con 0x7f4d88198e40 2023-07-14T17:37:34.109 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d8cb76700 1 -- 172.21.15.178:0/162235727 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8819fb40 con 0x7f4d88198e40 2023-07-14T17:37:34.110 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d777fe700 1 -- 172.21.15.178:0/162235727 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4d78004390 con 0x7f4d88198e40 2023-07-14T17:37:34.110 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d8cb76700 1 -- 172.21.15.178:0/162235727 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d881a0090 con 0x7f4d88198e40 2023-07-14T17:37:34.110 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.118+0000 7f4d777fe700 1 -- 172.21.15.178:0/162235727 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d78011a60 con 0x7f4d88198e40 2023-07-14T17:37:34.110 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.119+0000 7f4d777fe700 1 -- 172.21.15.178:0/162235727 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f4d78011cc0 con 0x7f4d88198e40 2023-07-14T17:37:34.110 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.119+0000 7f4d8cb76700 1 -- 172.21.15.178:0/162235727 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d88061c70 con 0x7f4d88198e40 2023-07-14T17:37:34.110 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.119+0000 7f4d777fe700 1 --2- 172.21.15.178:0/162235727 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4d68038c30 0x7f4d6803b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:34.110 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.120+0000 7f4d777fe700 1 -- 172.21.15.178:0/162235727 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4d7804add0 con 0x7f4d88198e40 2023-07-14T17:37:34.110 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.120+0000 7f4d8659c700 1 --2- 172.21.15.178:0/162235727 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4d68038c30 0x7f4d6803b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:34.111 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.120+0000 7f4d8659c700 1 --2- 172.21.15.178:0/162235727 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4d68038c30 0x7f4d6803b0e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c01efd0 tx=0x7f4d7c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:34.111 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.124+0000 7f4d777fe700 1 -- 172.21.15.178:0/162235727 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4d7801ae40 con 0x7f4d88198e40 2023-07-14T17:37:34.397 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.411+0000 7f4d8cb76700 1 -- 172.21.15.178:0/162235727 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d88069350 con 0x7f4d88198e40 2023-07-14T17:37:34.398 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.411+0000 7f4d777fe700 1 -- 172.21.15.178:0/162235727 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1274 (secure 0 0 0) 0x7f4d7800f2e0 con 0x7f4d88198e40 2023-07-14T17:37:34.399 INFO:teuthology.orchestra.run.smithi178.stdout: 2023-07-14T17:37:34.399 INFO:teuthology.orchestra.run.smithi178.stdout:{"epoch":3,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","modified":"2023-07-14T17:37:28.400828Z","created":"2023-07-14T17:35:46.715826Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:3300","nonce":0},{"type":"v1","addr":"172.21.15.178:6789","nonce":0}]},"addr":"172.21.15.178:6789/0","public_addr":"172.21.15.178:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-14T17:37:34.400 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.413+0000 7f4d8cb76700 1 -- 172.21.15.178:0/162235727 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4d68038c30 msgr2=0x7f4d6803b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:34.400 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.413+0000 7f4d8cb76700 1 --2- 172.21.15.178:0/162235727 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4d68038c30 0x7f4d6803b0e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c01efd0 tx=0x7f4d7c005960 comp rx=0 tx=0).stop 2023-07-14T17:37:34.401 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.413+0000 7f4d8cb76700 1 -- 172.21.15.178:0/162235727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d88198e40 msgr2=0x7f4d88192980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:34.401 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.413+0000 7f4d8cb76700 1 --2- 172.21.15.178:0/162235727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d88198e40 0x7f4d88192980 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4d7800b780 tx=0x7f4d7800bb40 comp rx=0 tx=0).stop 2023-07-14T17:37:34.401 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.413+0000 7f4d8cb76700 1 -- 172.21.15.178:0/162235727 shutdown_connections 2023-07-14T17:37:34.401 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.413+0000 7f4d8cb76700 1 --2- 172.21.15.178:0/162235727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4d88192ec0 0x7f4d88193360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:34.402 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.413+0000 7f4d8cb76700 1 --2- 172.21.15.178:0/162235727 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4d700057d0 0x7f4d88198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:34.402 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.413+0000 7f4d8cb76700 1 --2- 172.21.15.178:0/162235727 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4d68038c30 0x7f4d6803b0e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:34.402 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.413+0000 7f4d8cb76700 1 --2- 172.21.15.178:0/162235727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d88198e40 0x7f4d88192980 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:34.403 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.413+0000 7f4d8cb76700 1 -- 172.21.15.178:0/162235727 >> 172.21.15.178:0/162235727 conn(0x7f4d880fa260 msgr2=0x7f4d88100320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:34.403 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.413+0000 7f4d8cb76700 1 -- 172.21.15.178:0/162235727 shutdown_connections 2023-07-14T17:37:34.403 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:34.413+0000 7f4d8cb76700 1 -- 172.21.15.178:0/162235727 wait complete. 2023-07-14T17:37:34.403 INFO:teuthology.orchestra.run.smithi178.stderr:dumped monmap epoch 3 2023-07-14T17:37:34.788 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:34.788 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:34.789 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:37:34.789 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:34 smithi178 ceph-mon[129368]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:34.789 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:34.789 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:34.789 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:34 smithi178 ceph-mon[129368]: from='client.? 172.21.15.178:0/162235727' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T17:37:34.795 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-14T17:37:34.796 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph config generate-minimal-conf 2023-07-14T17:37:34.827 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:34.827 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:34.827 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:37:34.828 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:34 smithi088 ceph-mon[127465]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:34.828 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:34.828 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:34.828 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:34 smithi088 ceph-mon[127465]: from='client.? 172.21.15.178:0/162235727' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T17:37:35.362 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:37:35.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T17:37:35.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: mon.a calling monitor election 2023-07-14T17:37:35.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: mon.c calling monitor election 2023-07-14T17:37:35.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:35.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:37:35.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:35.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:35.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:35.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:35.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:35.730 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:35.730 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:35.730 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T17:37:35.730 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: monmap e3: 3 mons at {a=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],b=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} 2023-07-14T17:37:35.730 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: fsmap 2023-07-14T17:37:35.730 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T17:37:35.730 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: mgrmap e13: x(active, since 49s) 2023-07-14T17:37:35.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-14T17:37:35.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-14T17:37:35.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-14T17:37:35.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: mon.b (rank 2) addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] is down (out of quorum) 2023-07-14T17:37:35.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:35.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:35.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:35.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:35.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:37:35.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:35.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:35.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:35.733 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:35 smithi107 ceph-mon[128999]: from='client.? 172.21.15.178:0/162235727' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T17:37:36.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.564+0000 7f659a47e700 1 -- 172.21.15.88:0/2092273187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6594104660 msgr2=0x7f6594104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:36.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.565+0000 7f659a47e700 1 --2- 172.21.15.88:0/2092273187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6594104660 0x7f6594104a30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6584009910 tx=0x7f6584009c20 comp rx=0 tx=0).stop 2023-07-14T17:37:36.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.565+0000 7f659a47e700 1 -- 172.21.15.88:0/2092273187 shutdown_connections 2023-07-14T17:37:36.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.565+0000 7f659a47e700 1 --2- 172.21.15.88:0/2092273187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6594104660 0x7f6594104a30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:36.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.565+0000 7f659a47e700 1 -- 172.21.15.88:0/2092273187 >> 172.21.15.88:0/2092273187 conn(0x7f65940743e0 msgr2=0x7f65940747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:36.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.565+0000 7f659a47e700 1 -- 172.21.15.88:0/2092273187 shutdown_connections 2023-07-14T17:37:36.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.565+0000 7f659a47e700 1 -- 172.21.15.88:0/2092273187 wait complete. 2023-07-14T17:37:36.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.565+0000 7f659a47e700 1 Processor -- start 2023-07-14T17:37:36.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.565+0000 7f659a47e700 1 -- start start 2023-07-14T17:37:36.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f659a47e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6594104660 0x7f6594195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:36.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f659a47e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6594195fd0 0x7f6594196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:36.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f659a47e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f659419eb00 0x7f65941a0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:36.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f659a47e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65941a1620 con 0x7f6594195fd0 2023-07-14T17:37:36.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f659a47e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65941a1790 con 0x7f659419eb00 2023-07-14T17:37:36.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f659a47e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65941a1950 con 0x7f6594104660 2023-07-14T17:37:36.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f65937fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6594195fd0 0x7f6594196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:36.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f65937fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6594195fd0 0x7f6594196440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40532/0 (socket says 172.21.15.88:40532) 2023-07-14T17:37:36.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f65937fe700 1 -- 172.21.15.88:0/631202899 learned_addr learned my addr 172.21.15.88:0/631202899 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:37:36.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f6593fff700 1 --2- 172.21.15.88:0/631202899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6594104660 0x7f6594195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:36.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f65937fe700 1 -- 172.21.15.88:0/631202899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6594104660 msgr2=0x7f6594195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:36.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f65937fe700 1 --2- 172.21.15.88:0/631202899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6594104660 0x7f6594195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:36.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f65937fe700 1 -- 172.21.15.88:0/631202899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f659419eb00 msgr2=0x7f65941a0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:36.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f6598a1b700 1 --2- 172.21.15.88:0/631202899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f659419eb00 0x7f65941a0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:36.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f65937fe700 1 --2- 172.21.15.88:0/631202899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f659419eb00 0x7f65941a0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:36.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f65937fe700 1 -- 172.21.15.88:0/631202899 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65840095f0 con 0x7f6594195fd0 2023-07-14T17:37:36.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f6593fff700 1 --2- 172.21.15.88:0/631202899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6594104660 0x7f6594195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:37:36.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f65937fe700 1 --2- 172.21.15.88:0/631202899 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6594195fd0 0x7f6594196440 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f658800b780 tx=0x7f658800bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:36.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f65917fa700 1 -- 172.21.15.88:0/631202899 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6588004230 con 0x7f6594195fd0 2023-07-14T17:37:36.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f65917fa700 1 -- 172.21.15.88:0/631202899 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6588004390 con 0x7f6594195fd0 2023-07-14T17:37:36.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f659a47e700 1 -- 172.21.15.88:0/631202899 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65941a1c30 con 0x7f6594195fd0 2023-07-14T17:37:36.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f65917fa700 1 -- 172.21.15.88:0/631202899 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6588011a60 con 0x7f6594195fd0 2023-07-14T17:37:36.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.566+0000 7f659a47e700 1 -- 172.21.15.88:0/631202899 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65941a21b0 con 0x7f6594195fd0 2023-07-14T17:37:36.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.567+0000 7f65917fa700 1 -- 172.21.15.88:0/631202899 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f6588011c00 con 0x7f6594195fd0 2023-07-14T17:37:36.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.567+0000 7f659a47e700 1 -- 172.21.15.88:0/631202899 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6594061c70 con 0x7f6594195fd0 2023-07-14T17:37:36.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.567+0000 7f65917fa700 1 --2- 172.21.15.88:0/631202899 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f657c038c80 0x7f657c03b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:36.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.567+0000 7f65917fa700 1 -- 172.21.15.88:0/631202899 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f658804af20 con 0x7f6594195fd0 2023-07-14T17:37:36.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.568+0000 7f6593fff700 1 --2- 172.21.15.88:0/631202899 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f657c038c80 0x7f657c03b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:36.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.568+0000 7f6593fff700 1 --2- 172.21.15.88:0/631202899 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f657c038c80 0x7f657c03b130 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6584006010 tx=0x7f65840058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:36.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.572+0000 7f65917fa700 1 -- 172.21.15.88:0/631202899 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f658800f6d0 con 0x7f6594195fd0 2023-07-14T17:37:36.587 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: mon.b calling monitor election 2023-07-14T17:37:36.588 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: Updating smithi088:/etc/ceph/ceph.conf 2023-07-14T17:37:36.588 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: Updating smithi107:/etc/ceph/ceph.conf 2023-07-14T17:37:36.588 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: Updating smithi178:/etc/ceph/ceph.conf 2023-07-14T17:37:36.588 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: mon.b calling monitor election 2023-07-14T17:37:36.588 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: mon.c calling monitor election 2023-07-14T17:37:36.588 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: mon.a calling monitor election 2023-07-14T17:37:36.589 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T17:37:36.589 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: monmap e3: 3 mons at {a=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],b=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} 2023-07-14T17:37:36.589 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: fsmap 2023-07-14T17:37:36.589 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T17:37:36.589 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: mgrmap e13: x(active, since 51s) 2023-07-14T17:37:36.589 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-14T17:37:36.589 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: Cluster is now healthy 2023-07-14T17:37:36.590 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: overall HEALTH_OK 2023-07-14T17:37:36.590 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: Updating smithi107:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T17:37:36.590 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: Updating smithi088:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T17:37:36.590 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: Updating smithi178:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T17:37:36.590 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.590 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.590 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.591 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:36.591 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.591 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.591 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.591 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.591 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T17:37:36.591 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T17:37:36.591 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T17:37:36.592 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:36.592 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:36 smithi088 ceph-mon[127465]: Reconfiguring daemon mon.a on smithi088 2023-07-14T17:37:36.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: mon.b calling monitor election 2023-07-14T17:37:36.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: Updating smithi088:/etc/ceph/ceph.conf 2023-07-14T17:37:36.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: Updating smithi107:/etc/ceph/ceph.conf 2023-07-14T17:37:36.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: Updating smithi178:/etc/ceph/ceph.conf 2023-07-14T17:37:36.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: mon.b calling monitor election 2023-07-14T17:37:36.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: mon.c calling monitor election 2023-07-14T17:37:36.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: mon.a calling monitor election 2023-07-14T17:37:36.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T17:37:36.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: monmap e3: 3 mons at {a=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],b=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} 2023-07-14T17:37:36.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: fsmap 2023-07-14T17:37:36.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T17:37:36.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: mgrmap e13: x(active, since 51s) 2023-07-14T17:37:36.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-14T17:37:36.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: Cluster is now healthy 2023-07-14T17:37:36.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: overall HEALTH_OK 2023-07-14T17:37:36.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: Updating smithi107:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T17:37:36.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: Updating smithi088:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T17:37:36.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: Updating smithi178:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T17:37:36.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.671 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.671 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.671 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:36.671 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.671 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.671 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.671 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.671 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T17:37:36.672 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T17:37:36.672 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T17:37:36.672 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:36.672 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:36 smithi178 ceph-mon[129368]: Reconfiguring daemon mon.a on smithi088 2023-07-14T17:37:36.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: mon.b calling monitor election 2023-07-14T17:37:36.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: Updating smithi088:/etc/ceph/ceph.conf 2023-07-14T17:37:36.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: Updating smithi107:/etc/ceph/ceph.conf 2023-07-14T17:37:36.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: Updating smithi178:/etc/ceph/ceph.conf 2023-07-14T17:37:36.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: mon.b calling monitor election 2023-07-14T17:37:36.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: mon.c calling monitor election 2023-07-14T17:37:36.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: mon.a calling monitor election 2023-07-14T17:37:36.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T17:37:36.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: monmap e3: 3 mons at {a=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],b=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],c=[v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0]} removed_ranks: {} 2023-07-14T17:37:36.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: fsmap 2023-07-14T17:37:36.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T17:37:36.730 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: mgrmap e13: x(active, since 51s) 2023-07-14T17:37:36.730 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-14T17:37:36.730 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: Cluster is now healthy 2023-07-14T17:37:36.730 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: overall HEALTH_OK 2023-07-14T17:37:36.730 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: Updating smithi107:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T17:37:36.730 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: Updating smithi088:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T17:37:36.730 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: Updating smithi178:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T17:37:36.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:36.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:36.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T17:37:36.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T17:37:36.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T17:37:36.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:36.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:36 smithi107 ceph-mon[128999]: Reconfiguring daemon mon.a on smithi088 2023-07-14T17:37:36.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.771+0000 7f659a47e700 1 -- 172.21.15.88:0/631202899 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f6594067e30 con 0x7f6594195fd0 2023-07-14T17:37:36.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.771+0000 7f65917fa700 1 -- 172.21.15.88:0/631202899 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f658800f2e0 con 0x7f6594195fd0 2023-07-14T17:37:36.759 INFO:teuthology.orchestra.run.smithi088.stdout:# minimal ceph.conf for b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:37:36.759 INFO:teuthology.orchestra.run.smithi088.stdout:[global] 2023-07-14T17:37:36.759 INFO:teuthology.orchestra.run.smithi088.stdout: fsid = b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:37:36.759 INFO:teuthology.orchestra.run.smithi088.stdout: mon_host = [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] 2023-07-14T17:37:36.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.773+0000 7f659a47e700 1 -- 172.21.15.88:0/631202899 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f657c038c80 msgr2=0x7f657c03b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:36.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.773+0000 7f659a47e700 1 --2- 172.21.15.88:0/631202899 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f657c038c80 0x7f657c03b130 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6584006010 tx=0x7f65840058e0 comp rx=0 tx=0).stop 2023-07-14T17:37:36.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.773+0000 7f659a47e700 1 -- 172.21.15.88:0/631202899 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6594195fd0 msgr2=0x7f6594196440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:36.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.773+0000 7f659a47e700 1 --2- 172.21.15.88:0/631202899 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6594195fd0 0x7f6594196440 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f658800b780 tx=0x7f658800bb40 comp rx=0 tx=0).stop 2023-07-14T17:37:36.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.774+0000 7f659a47e700 1 -- 172.21.15.88:0/631202899 shutdown_connections 2023-07-14T17:37:36.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.774+0000 7f659a47e700 1 --2- 172.21.15.88:0/631202899 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f659419eb00 0x7f65941a0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:36.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.774+0000 7f659a47e700 1 --2- 172.21.15.88:0/631202899 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6594104660 0x7f6594195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:36.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.774+0000 7f659a47e700 1 --2- 172.21.15.88:0/631202899 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f657c038c80 0x7f657c03b130 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:36.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.774+0000 7f659a47e700 1 --2- 172.21.15.88:0/631202899 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6594195fd0 0x7f6594196440 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:36.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.774+0000 7f659a47e700 1 -- 172.21.15.88:0/631202899 >> 172.21.15.88:0/631202899 conn(0x7f65940743e0 msgr2=0x7f6594105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:36.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.774+0000 7f659a47e700 1 -- 172.21.15.88:0/631202899 shutdown_connections 2023-07-14T17:37:36.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:36.774+0000 7f659a47e700 1 -- 172.21.15.88:0/631202899 wait complete. 2023-07-14T17:37:37.309 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-14T17:37:37.309 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:37:37.310 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T17:37:37.377 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:37:37.378 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T17:37:37.443 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-07-14T17:37:37.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T17:37:37.480 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-07-14T17:37:37.481 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T17:37:37.547 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:37 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:37.548 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:37 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/631202899' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:37.551 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-07-14T17:37:37.551 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T17:37:37.582 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:37 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:37.583 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:37 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/631202899' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:37.588 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-07-14T17:37:37.588 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T17:37:37.657 INFO:tasks.cephadm:Adding mgr.x on smithi088 2023-07-14T17:37:37.658 INFO:tasks.cephadm:Adding mgr.y on smithi107 2023-07-14T17:37:37.658 INFO:tasks.cephadm:Adding mgr.z on smithi178 2023-07-14T17:37:37.658 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch apply mgr '3;smithi088=x;smithi107=y;smithi178=z' 2023-07-14T17:37:37.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:37 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:37:37.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:37 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/631202899' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:38.247 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T17:37:38.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:38 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:38.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:38 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:38.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:38 smithi178 ceph-mon[129368]: Reconfiguring mon.b (monmap changed)... 2023-07-14T17:37:38.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:38 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T17:37:38.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:38 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T17:37:38.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:38 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:38.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:38 smithi178 ceph-mon[129368]: Reconfiguring daemon mon.b on smithi107 2023-07-14T17:37:38.670 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:38 smithi178 ceph-mon[129368]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:38.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:38 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:38.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:38 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:38.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:38 smithi107 ceph-mon[128999]: Reconfiguring mon.b (monmap changed)... 2023-07-14T17:37:38.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:38 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T17:37:38.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:38 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T17:37:38.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:38 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:38.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:38 smithi107 ceph-mon[128999]: Reconfiguring daemon mon.b on smithi107 2023-07-14T17:37:38.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:38 smithi107 ceph-mon[128999]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:38.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:38 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:38.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:38 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:38.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:38 smithi088 ceph-mon[127465]: Reconfiguring mon.b (monmap changed)... 2023-07-14T17:37:38.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:38 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T17:37:38.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:38 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T17:37:38.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:38 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:38.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:38 smithi088 ceph-mon[127465]: Reconfiguring daemon mon.b on smithi107 2023-07-14T17:37:38.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:38 smithi088 ceph-mon[127465]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:39.643 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.653+0000 7f947afef700 1 -- 172.21.15.178:0/4120561583 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94740ff830 msgr2=0x7f94740ffc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:39.643 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.653+0000 7f947afef700 1 --2- 172.21.15.178:0/4120561583 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94740ff830 0x7f94740ffc40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9468009910 tx=0x7f9468009c20 comp rx=0 tx=0).stop 2023-07-14T17:37:39.643 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.653+0000 7f947afef700 1 -- 172.21.15.178:0/4120561583 shutdown_connections 2023-07-14T17:37:39.643 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.653+0000 7f947afef700 1 --2- 172.21.15.178:0/4120561583 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94740ff830 0x7f94740ffc40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:39.644 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.653+0000 7f947afef700 1 -- 172.21.15.178:0/4120561583 >> 172.21.15.178:0/4120561583 conn(0x7f94740fae00 msgr2=0x7f94740fd210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:39.644 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.653+0000 7f947afef700 1 -- 172.21.15.178:0/4120561583 shutdown_connections 2023-07-14T17:37:39.644 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.653+0000 7f947afef700 1 -- 172.21.15.178:0/4120561583 wait complete. 2023-07-14T17:37:39.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.653+0000 7f947afef700 1 Processor -- start 2023-07-14T17:37:39.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.653+0000 7f947afef700 1 -- start start 2023-07-14T17:37:39.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.654+0000 7f947afef700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94740ff830 0x7f94740743e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:39.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.654+0000 7f947afef700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9474072a30 0x7f9474072ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:39.645 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.654+0000 7f947afef700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94740733e0 0x7f947419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:39.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.654+0000 7f947afef700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947419f0a0 con 0x7f94740733e0 2023-07-14T17:37:39.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.654+0000 7f947afef700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947419f1e0 con 0x7f94740ff830 2023-07-14T17:37:39.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.654+0000 7f947afef700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947419f320 con 0x7f9474072a30 2023-07-14T17:37:39.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.654+0000 7f9473fff700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9474072a30 0x7f9474072ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:39.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.654+0000 7f9473fff700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9474072a30 0x7f9474072ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:54618/0 (socket says 172.21.15.178:54618) 2023-07-14T17:37:39.646 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.654+0000 7f9473fff700 1 -- 172.21.15.178:0/1126934708 learned_addr learned my addr 172.21.15.178:0/1126934708 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-07-14T17:37:39.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.654+0000 7f947958c700 1 --2- 172.21.15.178:0/1126934708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94740733e0 0x7f947419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:39.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.654+0000 7f9478d8b700 1 --2- 172.21.15.178:0/1126934708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94740ff830 0x7f94740743e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:39.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.654+0000 7f9473fff700 1 -- 172.21.15.178:0/1126934708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94740ff830 msgr2=0x7f94740743e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:39.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.654+0000 7f9473fff700 1 --2- 172.21.15.178:0/1126934708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94740ff830 0x7f94740743e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:39.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.654+0000 7f9473fff700 1 -- 172.21.15.178:0/1126934708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94740733e0 msgr2=0x7f947419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:39.647 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.654+0000 7f9473fff700 1 --2- 172.21.15.178:0/1126934708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94740733e0 0x7f947419eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:39.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.654+0000 7f9473fff700 1 -- 172.21.15.178:0/1126934708 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94680095f0 con 0x7f9474072a30 2023-07-14T17:37:39.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.654+0000 7f9478d8b700 1 --2- 172.21.15.178:0/1126934708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94740ff830 0x7f94740743e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:37:39.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.654+0000 7f9473fff700 1 --2- 172.21.15.178:0/1126934708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9474072a30 0x7f9474072ea0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f946000b780 tx=0x7f946000ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:39.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.655+0000 7f9471ffb700 1 -- 172.21.15.178:0/1126934708 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9460004230 con 0x7f9474072a30 2023-07-14T17:37:39.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.655+0000 7f9471ffb700 1 -- 172.21.15.178:0/1126934708 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9460004390 con 0x7f9474072a30 2023-07-14T17:37:39.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.655+0000 7f947afef700 1 -- 172.21.15.178:0/1126934708 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f947419f600 con 0x7f9474072a30 2023-07-14T17:37:39.648 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.655+0000 7f9471ffb700 1 -- 172.21.15.178:0/1126934708 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9460011a50 con 0x7f9474072a30 2023-07-14T17:37:39.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.655+0000 7f947afef700 1 -- 172.21.15.178:0/1126934708 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f947419fb80 con 0x7f9474072a30 2023-07-14T17:37:39.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.655+0000 7f9471ffb700 1 -- 172.21.15.178:0/1126934708 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f9460011bb0 con 0x7f9474072a30 2023-07-14T17:37:39.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.656+0000 7f9471ffb700 1 --2- 172.21.15.178:0/1126934708 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9464038c30 0x7f946403b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:39.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.656+0000 7f9471ffb700 1 -- 172.21.15.178:0/1126934708 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f946004bb70 con 0x7f9474072a30 2023-07-14T17:37:39.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.656+0000 7f9478d8b700 1 --2- 172.21.15.178:0/1126934708 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9464038c30 0x7f946403b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:39.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.656+0000 7f947afef700 1 -- 172.21.15.178:0/1126934708 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f947418e260 con 0x7f9474072a30 2023-07-14T17:37:39.649 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.657+0000 7f9478d8b700 1 --2- 172.21.15.178:0/1126934708 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9464038c30 0x7f946403b0e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9468005090 tx=0x7f946800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:39.650 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.662+0000 7f9471ffb700 1 -- 172.21.15.178:0/1126934708 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f946001a400 con 0x7f9474072a30 2023-07-14T17:37:39.857 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.871+0000 7f947afef700 1 -- 172.21.15.178:0/1126934708 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi088=x;smithi107=y;smithi178=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f9474060a80 con 0x7f9464038c30 2023-07-14T17:37:39.863 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.877+0000 7f9471ffb700 1 -- 172.21.15.178:0/1126934708 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9474060a80 con 0x7f9464038c30 2023-07-14T17:37:39.864 INFO:teuthology.orchestra.run.smithi178.stdout:Scheduled mgr update... 2023-07-14T17:37:39.866 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.880+0000 7f947afef700 1 -- 172.21.15.178:0/1126934708 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9464038c30 msgr2=0x7f946403b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:39.866 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.880+0000 7f947afef700 1 --2- 172.21.15.178:0/1126934708 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9464038c30 0x7f946403b0e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9468005090 tx=0x7f946800b540 comp rx=0 tx=0).stop 2023-07-14T17:37:39.866 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.880+0000 7f947afef700 1 -- 172.21.15.178:0/1126934708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9474072a30 msgr2=0x7f9474072ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:39.866 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.880+0000 7f947afef700 1 --2- 172.21.15.178:0/1126934708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9474072a30 0x7f9474072ea0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f946000b780 tx=0x7f946000ba90 comp rx=0 tx=0).stop 2023-07-14T17:37:39.866 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.880+0000 7f947afef700 1 -- 172.21.15.178:0/1126934708 shutdown_connections 2023-07-14T17:37:39.866 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.880+0000 7f947afef700 1 --2- 172.21.15.178:0/1126934708 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9464038c30 0x7f946403b0e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:39.867 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.880+0000 7f947afef700 1 --2- 172.21.15.178:0/1126934708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f94740733e0 0x7f947419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:39.867 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.880+0000 7f947afef700 1 --2- 172.21.15.178:0/1126934708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f94740ff830 0x7f94740743e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:39.867 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.880+0000 7f947afef700 1 --2- 172.21.15.178:0/1126934708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9474072a30 0x7f9474072ea0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:39.867 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.880+0000 7f947afef700 1 -- 172.21.15.178:0/1126934708 >> 172.21.15.178:0/1126934708 conn(0x7f94740fae00 msgr2=0x7f94740fbc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:39.867 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.880+0000 7f947afef700 1 -- 172.21.15.178:0/1126934708 shutdown_connections 2023-07-14T17:37:39.867 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:37:39.880+0000 7f947afef700 1 -- 172.21.15.178:0/1126934708 wait complete. 2023-07-14T17:37:39.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:39 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:39.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:39 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:39.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:39 smithi178 ceph-mon[129368]: Reconfiguring mon.c (monmap changed)... 2023-07-14T17:37:39.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:39 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T17:37:39.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:39 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T17:37:39.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:39 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:39.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:39 smithi178 ceph-mon[129368]: Reconfiguring daemon mon.c on smithi178 2023-07-14T17:37:40.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:39 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:40.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:39 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:40.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:39 smithi088 ceph-mon[127465]: Reconfiguring mon.c (monmap changed)... 2023-07-14T17:37:40.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:39 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T17:37:40.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:39 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T17:37:40.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:39 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:40.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:39 smithi088 ceph-mon[127465]: Reconfiguring daemon mon.c on smithi178 2023-07-14T17:37:40.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:39 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:40.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:39 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:40.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:39 smithi107 ceph-mon[128999]: Reconfiguring mon.c (monmap changed)... 2023-07-14T17:37:40.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:39 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T17:37:40.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:39 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T17:37:40.230 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:39 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:40.230 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:39 smithi107 ceph-mon[128999]: Reconfiguring daemon mon.c on smithi178 2023-07-14T17:37:40.399 DEBUG:teuthology.orchestra.run.smithi107:mgr.y> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mgr.y.service 2023-07-14T17:37:40.401 DEBUG:teuthology.orchestra.run.smithi178:mgr.z> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mgr.z.service 2023-07-14T17:37:40.403 INFO:tasks.cephadm:Deploying OSDs... 2023-07-14T17:37:40.404 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:37:40.404 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T17:37:40.424 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T17:37:40.425 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_1 2023-07-14T17:37:40.433 INFO:journalctl@ceph.mgr.z.smithi178.stdout:-- Logs begin at Fri 2023-07-14 17:14:18 UTC. -- 2023-07-14T17:37:40.435 INFO:journalctl@ceph.mgr.y.smithi107.stdout:-- Logs begin at Fri 2023-07-14 17:14:00 UTC. -- 2023-07-14T17:37:40.484 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T17:37:40.485 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T17:37:40.485 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 380648 Links: 1 2023-07-14T17:37:40.485 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T17:37:40.485 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T17:37:40.485 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-07-14 17:36:53.480623837 +0000 2023-07-14T17:37:40.486 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-07-14 17:28:10.356071606 +0000 2023-07-14T17:37:40.486 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-07-14 17:28:10.356071606 +0000 2023-07-14T17:37:40.486 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-07-14T17:37:40.486 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T17:37:40.553 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-07-14T17:37:40.554 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-07-14T17:37:40.554 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000250504 s, 2.0 MB/s 2023-07-14T17:37:40.555 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T17:37:40.614 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_2 2023-07-14T17:37:40.672 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T17:37:40.672 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T17:37:40.672 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 382039 Links: 1 2023-07-14T17:37:40.673 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T17:37:40.673 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T17:37:40.673 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-07-14 17:36:53.480623837 +0000 2023-07-14T17:37:40.674 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-07-14 17:28:10.883061396 +0000 2023-07-14T17:37:40.674 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-07-14 17:28:10.883061396 +0000 2023-07-14T17:37:40.674 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-07-14T17:37:40.674 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T17:37:40.738 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-07-14T17:37:40.738 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-07-14T17:37:40.738 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.00023129 s, 2.2 MB/s 2023-07-14T17:37:40.739 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T17:37:40.797 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_3 2023-07-14T17:37:40.854 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T17:37:40.854 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T17:37:40.854 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 376728 Links: 1 2023-07-14T17:37:40.854 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T17:37:40.854 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T17:37:40.855 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-07-14 17:36:53.480623837 +0000 2023-07-14T17:37:40.855 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-07-14 17:28:11.413051128 +0000 2023-07-14T17:37:40.855 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-07-14 17:28:11.413051128 +0000 2023-07-14T17:37:40.855 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-07-14T17:37:40.855 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T17:37:40.920 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-07-14T17:37:40.920 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-07-14T17:37:40.920 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000225761 s, 2.3 MB/s 2023-07-14T17:37:40.921 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T17:37:40.979 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_4 2023-07-14T17:37:41.036 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T17:37:41.036 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T17:37:41.036 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 378406 Links: 1 2023-07-14T17:37:41.037 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T17:37:41.037 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T17:37:41.037 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-07-14 17:36:53.480623837 +0000 2023-07-14T17:37:41.037 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-07-14 17:28:11.938040956 +0000 2023-07-14T17:37:41.037 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-07-14 17:28:11.938040956 +0000 2023-07-14T17:37:41.037 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-07-14T17:37:41.038 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T17:37:41.101 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-07-14T17:37:41.101 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-07-14T17:37:41.102 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000196142 s, 2.6 MB/s 2023-07-14T17:37:41.102 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T17:37:41.160 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2023-07-14T17:37:41.160 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T17:37:41.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:40 smithi178 ceph-mon[129368]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:41.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:40 smithi178 ceph-mon[129368]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi088=x;smithi107=y;smithi178=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:37:41.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:40 smithi178 ceph-mon[129368]: Saving service mgr spec with placement smithi088=x;smithi107=y;smithi178=z;count:3 2023-07-14T17:37:41.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:41.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:41.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:41.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:37:41.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:41.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:37:41.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:41.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T17:37:41.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T17:37:41.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T17:37:41.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:41.181 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T17:37:41.182 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2023-07-14T17:37:41.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:40 smithi088 ceph-mon[127465]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:41.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:40 smithi088 ceph-mon[127465]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi088=x;smithi107=y;smithi178=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:37:41.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:40 smithi088 ceph-mon[127465]: Saving service mgr spec with placement smithi088=x;smithi107=y;smithi178=z;count:3 2023-07-14T17:37:41.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:41.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:41.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:41.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:37:41.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:41.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:37:41.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:41.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T17:37:41.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T17:37:41.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T17:37:41.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:41.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:40 smithi107 ceph-mon[128999]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:41.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:40 smithi107 ceph-mon[128999]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi088=x;smithi107=y;smithi178=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:37:41.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:40 smithi107 ceph-mon[128999]: Saving service mgr spec with placement smithi088=x;smithi107=y;smithi178=z;count:3 2023-07-14T17:37:41.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:41.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:41.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:41.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:37:41.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:41.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:37:41.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:41.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T17:37:41.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T17:37:41.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T17:37:41.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:41.242 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T17:37:41.242 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T17:37:41.242 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 383081 Links: 1 2023-07-14T17:37:41.243 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T17:37:41.243 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T17:37:41.243 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-07-14 17:37:27.820413392 +0000 2023-07-14T17:37:41.243 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-07-14 17:28:38.760242166 +0000 2023-07-14T17:37:41.243 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-07-14 17:28:38.760242166 +0000 2023-07-14T17:37:41.243 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2023-07-14T17:37:41.244 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T17:37:41.312 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-07-14T17:37:41.313 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-07-14T17:37:41.313 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.00028286 s, 1.8 MB/s 2023-07-14T17:37:41.314 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T17:37:41.374 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2023-07-14T17:37:41.433 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T17:37:41.433 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T17:37:41.434 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 376521 Links: 1 2023-07-14T17:37:41.434 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T17:37:41.434 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T17:37:41.434 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-07-14 17:37:27.820413392 +0000 2023-07-14T17:37:41.434 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-07-14 17:28:39.294233762 +0000 2023-07-14T17:37:41.434 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-07-14 17:28:39.294233762 +0000 2023-07-14T17:37:41.434 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2023-07-14T17:37:41.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T17:37:41.501 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-07-14T17:37:41.501 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-07-14T17:37:41.502 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000224237 s, 2.3 MB/s 2023-07-14T17:37:41.503 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T17:37:41.563 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2023-07-14T17:37:41.622 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T17:37:41.622 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T17:37:41.622 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 376583 Links: 1 2023-07-14T17:37:41.622 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T17:37:41.623 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T17:37:41.623 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-07-14 17:37:27.820413392 +0000 2023-07-14T17:37:41.623 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-07-14 17:28:39.813225593 +0000 2023-07-14T17:37:41.623 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-07-14 17:28:39.813225593 +0000 2023-07-14T17:37:41.623 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2023-07-14T17:37:41.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T17:37:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-07-14T17:37:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-07-14T17:37:41.690 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.00024564 s, 2.1 MB/s 2023-07-14T17:37:41.691 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T17:37:41.750 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2023-07-14T17:37:41.809 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T17:37:41.809 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T17:37:41.810 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 382317 Links: 1 2023-07-14T17:37:41.810 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T17:37:41.810 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T17:37:41.811 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2023-07-14 17:37:27.820413392 +0000 2023-07-14T17:37:41.811 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2023-07-14 17:28:40.319217629 +0000 2023-07-14T17:37:41.811 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2023-07-14 17:28:40.319217629 +0000 2023-07-14T17:37:41.811 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2023-07-14T17:37:41.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T17:37:41.875 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2023-07-14T17:37:41.876 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2023-07-14T17:37:41.876 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000210581 s, 2.4 MB/s 2023-07-14T17:37:41.877 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T17:37:41.935 DEBUG:teuthology.orchestra.run.smithi178:> set -ex 2023-07-14T17:37:41.936 DEBUG:teuthology.orchestra.run.smithi178:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T17:37:41.953 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T17:37:41.953 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_1 2023-07-14T17:37:41.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:41 smithi107 ceph-mon[128999]: Deploying daemon mgr.z on smithi178 2023-07-14T17:37:42.010 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T17:37:42.011 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T17:37:42.011 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 6h/6d Inode: 379812 Links: 1 2023-07-14T17:37:42.011 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T17:37:42.011 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T17:37:42.011 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2023-07-14 17:37:41.974078230 +0000 2023-07-14T17:37:42.011 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2023-07-14 17:30:23.244817607 +0000 2023-07-14T17:37:42.012 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2023-07-14 17:30:23.244817607 +0000 2023-07-14T17:37:42.012 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2023-07-14T17:37:42.012 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T17:37:42.128 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:41 smithi178 ceph-mon[129368]: Deploying daemon mgr.z on smithi178 2023-07-14T17:37:42.132 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2023-07-14T17:37:42.132 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2023-07-14T17:37:42.132 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000130474 s, 3.9 MB/s 2023-07-14T17:37:42.133 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T17:37:42.152 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_2 2023-07-14T17:37:42.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:41 smithi088 ceph-mon[127465]: Deploying daemon mgr.z on smithi178 2023-07-14T17:37:42.212 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T17:37:42.213 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T17:37:42.213 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 6h/6d Inode: 382044 Links: 1 2023-07-14T17:37:42.213 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T17:37:42.213 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T17:37:42.214 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2023-07-14 17:37:42.099076026 +0000 2023-07-14T17:37:42.214 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2023-07-14 17:30:23.754808615 +0000 2023-07-14T17:37:42.214 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2023-07-14 17:30:23.754808615 +0000 2023-07-14T17:37:42.214 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2023-07-14T17:37:42.214 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T17:37:42.280 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2023-07-14T17:37:42.280 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2023-07-14T17:37:42.280 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000106907 s, 4.8 MB/s 2023-07-14T17:37:42.281 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T17:37:42.341 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_3 2023-07-14T17:37:42.398 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T17:37:42.398 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T17:37:42.398 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 6h/6d Inode: 385150 Links: 1 2023-07-14T17:37:42.399 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T17:37:42.399 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T17:37:42.399 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2023-07-14 17:37:42.099076026 +0000 2023-07-14T17:37:42.400 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2023-07-14 17:30:24.262799658 +0000 2023-07-14T17:37:42.400 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2023-07-14 17:30:24.262799658 +0000 2023-07-14T17:37:42.400 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2023-07-14T17:37:42.401 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T17:37:42.417 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:42 smithi178 systemd[1]: Starting Ceph mgr.z for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:37:42.442 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2023-07-14T17:37:42.442 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2023-07-14T17:37:42.442 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000116577 s, 4.4 MB/s 2023-07-14T17:37:42.443 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T17:37:42.501 DEBUG:teuthology.orchestra.run.smithi178:> stat /dev/vg_nvme/lv_4 2023-07-14T17:37:42.559 INFO:teuthology.orchestra.run.smithi178.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T17:37:42.559 INFO:teuthology.orchestra.run.smithi178.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T17:37:42.559 INFO:teuthology.orchestra.run.smithi178.stdout:Device: 6h/6d Inode: 385307 Links: 1 2023-07-14T17:37:42.559 INFO:teuthology.orchestra.run.smithi178.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T17:37:42.560 INFO:teuthology.orchestra.run.smithi178.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T17:37:42.560 INFO:teuthology.orchestra.run.smithi178.stdout:Access: 2023-07-14 17:37:42.099076026 +0000 2023-07-14T17:37:42.560 INFO:teuthology.orchestra.run.smithi178.stdout:Modify: 2023-07-14 17:30:24.781790506 +0000 2023-07-14T17:37:42.560 INFO:teuthology.orchestra.run.smithi178.stdout:Change: 2023-07-14 17:30:24.781790506 +0000 2023-07-14T17:37:42.560 INFO:teuthology.orchestra.run.smithi178.stdout: Birth: - 2023-07-14T17:37:42.560 DEBUG:teuthology.orchestra.run.smithi178:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T17:37:42.624 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records in 2023-07-14T17:37:42.625 INFO:teuthology.orchestra.run.smithi178.stderr:1+0 records out 2023-07-14T17:37:42.625 INFO:teuthology.orchestra.run.smithi178.stderr:512 bytes copied, 0.000101764 s, 5.0 MB/s 2023-07-14T17:37:42.626 DEBUG:teuthology.orchestra.run.smithi178:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T17:37:42.683 INFO:tasks.cephadm:Deploying osd.0 on smithi088 with /dev/vg_nvme/lv_4... 2023-07-14T17:37:42.683 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T17:37:42.763 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:42 smithi178 podman[131325]: 2023-07-14T17:37:43.058 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:42 smithi178 ceph-mon[129368]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:43.059 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:42 smithi178 ceph-mgr[131510]: -- 172.21.15.178:0/1342868935 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563219b471e0 con 0x563219b1cc00 2023-07-14T17:37:43.059 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:42 smithi178 bash[131325]: 47bbe9518e207b0781539896e0e40eb6d48c547625fa5ab4d0f13fef3f80ebc5 2023-07-14T17:37:43.060 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:42 smithi178 systemd[1]: Started Ceph mgr.z for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:37:43.060 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:42 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:42.904+0000 7efed1df0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T17:37:43.060 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:42 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:42.984+0000 7efed1df0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T17:37:43.060 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:43 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:43.072+0000 7efed1df0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T17:37:43.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:42 smithi088 ceph-mon[127465]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:43.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:42 smithi107 ceph-mon[128999]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:43.245 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:37:43.315 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:43 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:43.163+0000 7efed1df0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T17:37:43.619 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:43 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:43.329+0000 7efed1df0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T17:37:43.887 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:43 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:43.634+0000 7efed1df0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T17:37:43.888 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:43 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:43.721+0000 7efed1df0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T17:37:43.888 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:43 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:43.848+0000 7efed1df0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T17:37:43.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:43.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:43.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:43.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T17:37:43.979 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T17:37:43.979 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T17:37:43.979 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:43.979 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:43 smithi107 ceph-mon[128999]: Deploying daemon mgr.y on smithi107 2023-07-14T17:37:44.167 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:43 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:43.942+0000 7efed1df0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T17:37:44.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:44.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:44.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:44.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T17:37:44.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T17:37:44.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T17:37:44.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:44.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:43 smithi178 ceph-mon[129368]: Deploying daemon mgr.y on smithi107 2023-07-14T17:37:44.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:44.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:44.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:44.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T17:37:44.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T17:37:44.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T17:37:44.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:44.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:43 smithi088 ceph-mon[127465]: Deploying daemon mgr.y on smithi107 2023-07-14T17:37:44.667 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:44 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:44.357+0000 7efed1df0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T17:37:44.712 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T17:37:44.712 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T17:37:44.712 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2023-07-14T17:37:44.712 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2023-07-14T17:37:44.713 INFO:teuthology.orchestra.run.smithi088.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0108567 s, 966 MB/s 2023-07-14T17:37:44.713 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2023-07-14T17:37:44.894 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:44 smithi107 systemd[1]: Starting Ceph mgr.y for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:37:45.130 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:44 smithi088 ceph-mon[127465]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:45.146 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:44 smithi107 ceph-mon[128999]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:45.146 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:44 smithi107 podman[130440]: 2023-07-14T17:37:45.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:44 smithi178 ceph-mon[129368]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:45.168 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:44 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:44.923+0000 7efed1df0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T17:37:45.168 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:45 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:45.093+0000 7efed1df0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T17:37:45.198 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi088:vg_nvme/lv_4 2023-07-14T17:37:45.446 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:45 smithi107 bash[130440]: e740b1f28d2aef7c2b9942f0c721d11eceab11c626fae7e74044fe37d07588f4 2023-07-14T17:37:45.446 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:45 smithi107 ceph-mgr[130480]: -- 172.21.15.107:0/751126185 <== mon.2 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560e1224b1e0 con 0x560e12221000 2023-07-14T17:37:45.447 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:45 smithi107 systemd[1]: Started Ceph mgr.y for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:37:45.447 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:45 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:45.285+0000 7f4b13a54200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T17:37:45.447 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:45 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:45.368+0000 7f4b13a54200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T17:37:45.447 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:45 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:45.461+0000 7f4b13a54200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T17:37:45.719 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:45 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:45.560+0000 7f4b13a54200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T17:37:45.752 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:37:45.977 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:45 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:45.734+0000 7f4b13a54200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T17:37:46.156 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:45 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:45.894+0000 7efed1df0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T17:37:46.156 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:45 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:45.988+0000 7efed1df0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T17:37:46.395 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:46 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:46.395 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:46 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:46.396 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:46 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:46.396 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:46 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:46.396 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:46 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:37:46.396 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:46 smithi107 ceph-mon[128999]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:46.397 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:46 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:46.075+0000 7f4b13a54200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T17:37:46.397 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:46 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:46.166+0000 7f4b13a54200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T17:37:46.397 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:46 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:46.301+0000 7f4b13a54200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T17:37:46.397 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:46 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:46.409+0000 7f4b13a54200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T17:37:46.417 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:46 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:46.170+0000 7efed1df0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T17:37:46.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:46 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:46.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:46 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:46.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:46 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:46.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:46 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:46.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:46 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:37:46.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:46 smithi178 ceph-mon[129368]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:46.498 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:46 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:46.499 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:46 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:46.499 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:46 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:46.499 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:46 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:46.499 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:46 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:37:46.499 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:46 smithi088 ceph-mon[127465]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:46.853 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:46 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:46.462+0000 7efed1df0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T17:37:46.853 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:46 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:46.704+0000 7efed1df0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T17:37:46.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.939+0000 7f772aaaf700 1 -- 172.21.15.88:0/2196160646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7724100240 msgr2=0x7f7724100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:46.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.939+0000 7f772aaaf700 1 --2- 172.21.15.88:0/2196160646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7724100240 0x7f7724100650 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7718009990 tx=0x7f7718009ca0 comp rx=0 tx=0).stop 2023-07-14T17:37:46.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.939+0000 7f772aaaf700 1 -- 172.21.15.88:0/2196160646 shutdown_connections 2023-07-14T17:37:46.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.939+0000 7f772aaaf700 1 --2- 172.21.15.88:0/2196160646 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f77240ff7c0 0x7f7724107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:46.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.939+0000 7f772aaaf700 1 --2- 172.21.15.88:0/2196160646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7724100240 0x7f7724100650 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:46.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.939+0000 7f772aaaf700 1 --2- 172.21.15.88:0/2196160646 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77240fee10 0x7f77240ff280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:46.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.939+0000 7f772aaaf700 1 -- 172.21.15.88:0/2196160646 >> 172.21.15.88:0/2196160646 conn(0x7f77240f9990 msgr2=0x7f77240fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:37:46.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.939+0000 7f772aaaf700 1 -- 172.21.15.88:0/2196160646 shutdown_connections 2023-07-14T17:37:46.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.939+0000 7f772aaaf700 1 -- 172.21.15.88:0/2196160646 wait complete. 2023-07-14T17:37:46.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f772aaaf700 1 Processor -- start 2023-07-14T17:37:46.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f772aaaf700 1 -- start start 2023-07-14T17:37:46.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f772aaaf700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f77240fee10 0x7f7724195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:46.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f772aaaf700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77240ff7c0 0x7f7724195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:46.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f772aaaf700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7724100240 0x7f772419cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:46.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f772aaaf700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772419d490 con 0x7f7724100240 2023-07-14T17:37:46.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f772aaaf700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772419d5d0 con 0x7f77240ff7c0 2023-07-14T17:37:46.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f772aaaf700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772419d760 con 0x7f77240fee10 2023-07-14T17:37:46.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f772904c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7724100240 0x7f772419cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:46.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f7723fff700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77240ff7c0 0x7f7724195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:46.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f772904c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7724100240 0x7f772419cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40544/0 (socket says 172.21.15.88:40544) 2023-07-14T17:37:46.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f772884b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f77240fee10 0x7f7724195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:46.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f772904c700 1 -- 172.21.15.88:0/2054774835 learned_addr learned my addr 172.21.15.88:0/2054774835 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:37:46.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f772904c700 1 -- 172.21.15.88:0/2054774835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f77240fee10 msgr2=0x7f7724195900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:46.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f772904c700 1 --2- 172.21.15.88:0/2054774835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f77240fee10 0x7f7724195900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:46.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f772904c700 1 -- 172.21.15.88:0/2054774835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77240ff7c0 msgr2=0x7f7724195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:37:46.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f772904c700 1 --2- 172.21.15.88:0/2054774835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77240ff7c0 0x7f7724195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:37:46.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f772904c700 1 -- 172.21.15.88:0/2054774835 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77180095f0 con 0x7f7724100240 2023-07-14T17:37:46.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.940+0000 7f7723fff700 1 --2- 172.21.15.88:0/2054774835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77240ff7c0 0x7f7724195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T17:37:46.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.941+0000 7f772904c700 1 --2- 172.21.15.88:0/2054774835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7724100240 0x7f772419cec0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f771c00ef60 tx=0x7f771c00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:46.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.941+0000 7f7721ffb700 1 -- 172.21.15.88:0/2054774835 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c018070 con 0x7f7724100240 2023-07-14T17:37:46.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.941+0000 7f7721ffb700 1 -- 172.21.15.88:0/2054774835 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f771c00f040 con 0x7f7724100240 2023-07-14T17:37:46.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.941+0000 7f772aaaf700 1 -- 172.21.15.88:0/2054774835 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f772419d9e0 con 0x7f7724100240 2023-07-14T17:37:46.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.941+0000 7f7721ffb700 1 -- 172.21.15.88:0/2054774835 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c00cc60 con 0x7f7724100240 2023-07-14T17:37:46.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.941+0000 7f772aaaf700 1 -- 172.21.15.88:0/2054774835 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f772419dee0 con 0x7f7724100240 2023-07-14T17:37:46.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.942+0000 7f7721ffb700 1 -- 172.21.15.88:0/2054774835 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f771c004000 con 0x7f7724100240 2023-07-14T17:37:46.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.942+0000 7f7721ffb700 1 --2- 172.21.15.88:0/2054774835 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f77140389e0 0x7f771403ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:37:46.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.942+0000 7f7721ffb700 1 -- 172.21.15.88:0/2054774835 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f771c04af10 con 0x7f7724100240 2023-07-14T17:37:46.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.942+0000 7f772884b700 1 --2- 172.21.15.88:0/2054774835 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f77140389e0 0x7f771403ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:37:46.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.942+0000 7f772aaaf700 1 -- 172.21.15.88:0/2054774835 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7724106de0 con 0x7f7724100240 2023-07-14T17:37:46.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.942+0000 7f772884b700 1 --2- 172.21.15.88:0/2054774835 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f77140389e0 0x7f771403ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7718006010 tx=0x7f771800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:37:46.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:46.947+0000 7f7721ffb700 1 -- 172.21.15.88:0/2054774835 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f771c00ce00 con 0x7f7724100240 2023-07-14T17:37:47.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:47.160+0000 7f772aaaf700 1 -- 172.21.15.88:0/2054774835 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f7724060a80 con 0x7f77140389e0 2023-07-14T17:37:47.227 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:46 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:46.862+0000 7f4b13a54200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T17:37:47.417 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:46 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:46.865+0000 7efed1df0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T17:37:47.418 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:47.133+0000 7efed1df0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T17:37:47.727 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:47.449+0000 7f4b13a54200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T17:37:47.728 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:47.617+0000 7f4b13a54200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T17:37:48.138 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:47.832+0000 7efed1df0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T17:37:48.139 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:47.963+0000 7efed1df0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T17:37:48.139 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:48 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:48.151+0000 7efed1df0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T17:37:48.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:48.139 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:48.140 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:48.140 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:48.140 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:48.140 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:37:48.140 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:48.140 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-mon[129368]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T17:37:48.140 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T17:37:48.141 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T17:37:48.141 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:48.141 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-mon[129368]: Reconfiguring daemon mgr.x on smithi088 2023-07-14T17:37:48.141 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-mon[129368]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:37:48.141 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:37:48.141 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:37:48.141 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:47 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:48.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:47 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:48.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:47 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:48.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:47 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:48.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:47 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:48.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:47 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:48.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:47 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:37:48.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:47 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:48.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:47 smithi088 ceph-mon[127465]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T17:37:48.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:47 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T17:37:48.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:47 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T17:37:48.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:47 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:48.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:47 smithi088 ceph-mon[127465]: Reconfiguring daemon mgr.x on smithi088 2023-07-14T17:37:48.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:47 smithi088 ceph-mon[127465]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:37:48.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:47 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:37:48.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:47 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:37:48.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:47 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:48.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:47 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:48.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:47 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:48.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:47 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:48.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:47 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:48.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:47 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:48.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:47 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:37:48.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:47 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:48.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:47 smithi107 ceph-mon[128999]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T17:37:48.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:47 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T17:37:48.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:47 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T17:37:48.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:47 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:48.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:47 smithi107 ceph-mon[128999]: Reconfiguring daemon mgr.x on smithi088 2023-07-14T17:37:48.230 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:47 smithi107 ceph-mon[128999]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:37:48.230 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:47 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:37:48.230 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:47 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:37:48.230 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:47 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:48.417 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:48 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:48.265+0000 7efed1df0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T17:37:48.623 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:48 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:48.377+0000 7f4b13a54200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T17:37:48.624 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:48 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:48.468+0000 7f4b13a54200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T17:37:48.898 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:48 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:48.638+0000 7f4b13a54200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T17:37:48.917 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:48 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:48.479+0000 7efed1df0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T17:37:49.227 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:48 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:48.912+0000 7f4b13a54200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T17:37:49.228 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:49 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:49.142+0000 7f4b13a54200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T17:37:49.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:49 smithi107 ceph-mon[128999]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:49.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:49.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:49.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:37:49.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:49.327 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:49 smithi178 ceph-mon[129368]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:49.327 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:49.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:49.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:37:49.328 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:49.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:49 smithi088 ceph-mon[127465]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:49.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:49.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:49.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:37:49.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:49.534 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:49 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:49.289+0000 7f4b13a54200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T17:37:49.667 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:37:49 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z[131424]: 2023-07-14T17:37:49.340+0000 7efed1df0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T17:37:49.977 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:49 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:49.548+0000 7f4b13a54200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T17:37:50.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:50.100+0000 7f7721ffb700 1 -- 172.21.15.88:0/2054774835 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mgrmap(e 14) v1 ==== 90599+0+0 (secure 0 0 0) 0x7f771c004350 con 0x7f7724100240 2023-07-14T17:37:50.357 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:50 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:50.242+0000 7f4b13a54200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T17:37:50.357 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:50 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:50.357 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:50 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:37:50.357 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:50 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:50.358 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:50 smithi107 ceph-mon[128999]: Standby manager daemon z started 2023-07-14T17:37:50.358 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:50 smithi107 ceph-mon[128999]: from='mgr.? 172.21.15.178:0/3785464343' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T17:37:50.358 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:50 smithi107 ceph-mon[128999]: from='mgr.? 172.21.15.178:0/3785464343' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T17:37:50.358 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:50 smithi107 ceph-mon[128999]: from='mgr.? 172.21.15.178:0/3785464343' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T17:37:50.358 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:50 smithi107 ceph-mon[128999]: from='mgr.? 172.21.15.178:0/3785464343' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T17:37:50.358 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:50 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2821486949' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32bbe236-4df9-42d6-8541-5d24d944eeee"}]: dispatch 2023-07-14T17:37:50.359 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:50 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2821486949' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32bbe236-4df9-42d6-8541-5d24d944eeee"}]': finished 2023-07-14T17:37:50.359 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:50 smithi107 ceph-mon[128999]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T17:37:50.359 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:50 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T17:37:50.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:50 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:50.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:50 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:37:50.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:50 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:50.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:50 smithi178 ceph-mon[129368]: Standby manager daemon z started 2023-07-14T17:37:50.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:50 smithi178 ceph-mon[129368]: from='mgr.? 172.21.15.178:0/3785464343' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T17:37:50.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:50 smithi178 ceph-mon[129368]: from='mgr.? 172.21.15.178:0/3785464343' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T17:37:50.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:50 smithi178 ceph-mon[129368]: from='mgr.? 172.21.15.178:0/3785464343' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T17:37:50.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:50 smithi178 ceph-mon[129368]: from='mgr.? 172.21.15.178:0/3785464343' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T17:37:50.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:50 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2821486949' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32bbe236-4df9-42d6-8541-5d24d944eeee"}]: dispatch 2023-07-14T17:37:50.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:50 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2821486949' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32bbe236-4df9-42d6-8541-5d24d944eeee"}]': finished 2023-07-14T17:37:50.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:50 smithi178 ceph-mon[129368]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T17:37:50.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:50 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T17:37:50.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:50 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:50.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:50 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:37:50.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:50 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:37:50.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:50 smithi088 ceph-mon[127465]: Standby manager daemon z started 2023-07-14T17:37:50.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:50 smithi088 ceph-mon[127465]: from='mgr.? 172.21.15.178:0/3785464343' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T17:37:50.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:50 smithi088 ceph-mon[127465]: from='mgr.? 172.21.15.178:0/3785464343' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T17:37:50.447 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:50 smithi088 ceph-mon[127465]: from='mgr.? 172.21.15.178:0/3785464343' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T17:37:50.447 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:50 smithi088 ceph-mon[127465]: from='mgr.? 172.21.15.178:0/3785464343' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T17:37:50.447 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:50 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2821486949' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32bbe236-4df9-42d6-8541-5d24d944eeee"}]: dispatch 2023-07-14T17:37:50.447 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:50 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2821486949' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32bbe236-4df9-42d6-8541-5d24d944eeee"}]': finished 2023-07-14T17:37:50.447 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:50 smithi088 ceph-mon[127465]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T17:37:50.448 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:50 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T17:37:50.660 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:50 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:50.371+0000 7f4b13a54200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T17:37:50.660 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:50 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:50.560+0000 7f4b13a54200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T17:37:50.977 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:50 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:50.674+0000 7f4b13a54200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T17:37:50.978 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:50 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:50.890+0000 7f4b13a54200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T17:37:51.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:51 smithi178 ceph-mon[129368]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:51.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:51 smithi178 ceph-mon[129368]: mgrmap e14: x(active, since 66s), standbys: z 2023-07-14T17:37:51.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:51 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T17:37:51.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:51 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1272782342' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:37:51.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:51 smithi088 ceph-mon[127465]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:51.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:51 smithi088 ceph-mon[127465]: mgrmap e14: x(active, since 66s), standbys: z 2023-07-14T17:37:51.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:51 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T17:37:51.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:51 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1272782342' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:37:51.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:51 smithi107 ceph-mon[128999]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:51.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:51 smithi107 ceph-mon[128999]: mgrmap e14: x(active, since 66s), standbys: z 2023-07-14T17:37:51.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:51 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T17:37:51.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:51 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1272782342' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:37:51.977 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:37:51 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y[130456]: 2023-07-14T17:37:51.738+0000 7f4b13a54200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T17:37:52.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:37:52.109+0000 7f7721ffb700 1 -- 172.21.15.88:0/2054774835 <== mon.0 v2:172.21.15.88:3300/0 8 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f771c0075f0 con 0x7f7724100240 2023-07-14T17:37:52.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:52 smithi178 ceph-mon[129368]: Standby manager daemon y started 2023-07-14T17:37:52.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:52 smithi178 ceph-mon[129368]: from='mgr.? 172.21.15.107:0/1485997054' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T17:37:52.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:52 smithi178 ceph-mon[129368]: from='mgr.? 172.21.15.107:0/1485997054' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T17:37:52.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:52 smithi178 ceph-mon[129368]: from='mgr.? 172.21.15.107:0/1485997054' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T17:37:52.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:52 smithi178 ceph-mon[129368]: from='mgr.? 172.21.15.107:0/1485997054' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T17:37:52.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:52 smithi088 ceph-mon[127465]: Standby manager daemon y started 2023-07-14T17:37:52.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:52 smithi088 ceph-mon[127465]: from='mgr.? 172.21.15.107:0/1485997054' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T17:37:52.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:52 smithi088 ceph-mon[127465]: from='mgr.? 172.21.15.107:0/1485997054' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T17:37:52.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:52 smithi088 ceph-mon[127465]: from='mgr.? 172.21.15.107:0/1485997054' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T17:37:52.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:52 smithi088 ceph-mon[127465]: from='mgr.? 172.21.15.107:0/1485997054' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T17:37:52.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:52 smithi107 ceph-mon[128999]: Standby manager daemon y started 2023-07-14T17:37:52.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:52 smithi107 ceph-mon[128999]: from='mgr.? 172.21.15.107:0/1485997054' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T17:37:52.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:52 smithi107 ceph-mon[128999]: from='mgr.? 172.21.15.107:0/1485997054' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T17:37:52.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:52 smithi107 ceph-mon[128999]: from='mgr.? 172.21.15.107:0/1485997054' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T17:37:52.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:52 smithi107 ceph-mon[128999]: from='mgr.? 172.21.15.107:0/1485997054' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T17:37:53.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:53 smithi178 ceph-mon[129368]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:53.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:53 smithi178 ceph-mon[129368]: mgrmap e15: x(active, since 68s), standbys: z, y 2023-07-14T17:37:53.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:53 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T17:37:53.433 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:53 smithi088 ceph-mon[127465]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:53.433 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:53 smithi088 ceph-mon[127465]: mgrmap e15: x(active, since 68s), standbys: z, y 2023-07-14T17:37:53.433 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:53 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T17:37:53.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:53 smithi107 ceph-mon[128999]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:53.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:53 smithi107 ceph-mon[128999]: mgrmap e15: x(active, since 68s), standbys: z, y 2023-07-14T17:37:53.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:53 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T17:37:55.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:55 smithi088 ceph-mon[127465]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:55.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:55 smithi178 ceph-mon[129368]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:55.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:55 smithi107 ceph-mon[128999]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:56.866 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:56 smithi088 ceph-mon[127465]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:56.867 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:56 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T17:37:56.867 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:56 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:56.867 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:56 smithi088 ceph-mon[127465]: Deploying daemon osd.0 on smithi088 2023-07-14T17:37:56.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:56 smithi178 ceph-mon[129368]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:56.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:56 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T17:37:56.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:56 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:56.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:56 smithi178 ceph-mon[129368]: Deploying daemon osd.0 on smithi088 2023-07-14T17:37:56.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:56 smithi107 ceph-mon[128999]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:56.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:56 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T17:37:56.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:56 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:37:56.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:56 smithi107 ceph-mon[128999]: Deploying daemon osd.0 on smithi088 2023-07-14T17:37:58.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:37:58 smithi088 ceph-mon[127465]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:59.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:37:58 smithi178 ceph-mon[129368]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:37:59.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:37:58 smithi107 ceph-mon[128999]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:38:01.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:00 smithi178 ceph-mon[129368]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:38:01.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:00 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:01.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:00 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:01.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:00 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:38:01.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:00 smithi088 ceph-mon[127465]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:38:01.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:00 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:01.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:00 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:01.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:00 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:38:01.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:00 smithi107 ceph-mon[128999]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:38:01.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:00 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:01.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:00 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:01.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:00 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:38:03.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:02 smithi178 ceph-mon[129368]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:38:03.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:02 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:03.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:02 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:03.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:02 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:03.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:02 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:38:03.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:02 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:03.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:02 smithi088 ceph-mon[127465]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:38:03.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:02 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:03.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:02 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:03.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:02 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:03.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:02 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:38:03.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:02 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:03.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:02 smithi107 ceph-mon[128999]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:38:03.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:02 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:03.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:02 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:03.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:02 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:03.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:02 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:38:03.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:02 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:03.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:03.365+0000 7f7721ffb700 1 -- 172.21.15.88:0/2054774835 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7724060a80 con 0x7f77140389e0 2023-07-14T17:38:03.351 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 0 on host 'smithi088' 2023-07-14T17:38:03.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:03.367+0000 7f772aaaf700 1 -- 172.21.15.88:0/2054774835 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f77140389e0 msgr2=0x7f771403ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:38:03.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:03.367+0000 7f772aaaf700 1 --2- 172.21.15.88:0/2054774835 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f77140389e0 0x7f771403ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7718006010 tx=0x7f771800b260 comp rx=0 tx=0).stop 2023-07-14T17:38:03.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:03.367+0000 7f772aaaf700 1 -- 172.21.15.88:0/2054774835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7724100240 msgr2=0x7f772419cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:38:03.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:03.367+0000 7f772aaaf700 1 --2- 172.21.15.88:0/2054774835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7724100240 0x7f772419cec0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f771c00ef60 tx=0x7f771c00ef90 comp rx=0 tx=0).stop 2023-07-14T17:38:03.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:03.367+0000 7f772aaaf700 1 -- 172.21.15.88:0/2054774835 shutdown_connections 2023-07-14T17:38:03.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:03.367+0000 7f772aaaf700 1 --2- 172.21.15.88:0/2054774835 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f77140389e0 0x7f771403ae90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:03.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:03.367+0000 7f772aaaf700 1 --2- 172.21.15.88:0/2054774835 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7724100240 0x7f772419cec0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:03.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:03.367+0000 7f772aaaf700 1 --2- 172.21.15.88:0/2054774835 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f77240fee10 0x7f7724195900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:03.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:03.367+0000 7f772aaaf700 1 --2- 172.21.15.88:0/2054774835 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f77240ff7c0 0x7f7724195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:03.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:03.367+0000 7f772aaaf700 1 -- 172.21.15.88:0/2054774835 >> 172.21.15.88:0/2054774835 conn(0x7f77240f9990 msgr2=0x7f77240fbcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:38:03.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:03.368+0000 7f772aaaf700 1 -- 172.21.15.88:0/2054774835 shutdown_connections 2023-07-14T17:38:03.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:03.368+0000 7f772aaaf700 1 -- 172.21.15.88:0/2054774835 wait complete. 2023-07-14T17:38:03.738 DEBUG:teuthology.orchestra.run.smithi088:osd.0> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.0.service 2023-07-14T17:38:03.741 INFO:tasks.cephadm:Deploying osd.1 on smithi107 with /dev/vg_nvme/lv_4... 2023-07-14T17:38:03.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T17:38:03.777 INFO:journalctl@ceph.osd.0.smithi088.stdout:-- Logs begin at Fri 2023-07-14 17:14:35 UTC. -- 2023-07-14T17:38:04.302 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T17:38:04.567 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:04 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:04.568 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:04 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:04.568 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:04 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:38:04.568 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:04 smithi107 ceph-mon[128999]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:38:04.592 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:04 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:04.593 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:04 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:04.593 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:04 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:38:04.593 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:04 smithi088 ceph-mon[127465]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:38:04.593 INFO:journalctl@ceph.osd.0.smithi088.stdout:Jul 14 17:38:04 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-0[140436]: 2023-07-14T17:38:04.518+0000 7fe36e80b640 -1 osd.0 0 log_to_monitors true 2023-07-14T17:38:04.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:04 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:04.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:04 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:04.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:04 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:38:04.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:04 smithi178 ceph-mon[129368]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:38:05.576 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:05 smithi088 ceph-mon[127465]: from='osd.0 [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T17:38:05.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:05 smithi178 ceph-mon[129368]: from='osd.0 [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T17:38:05.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:05 smithi107 ceph-mon[128999]: from='osd.0 [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T17:38:05.826 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T17:38:05.827 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T17:38:05.827 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2023-07-14T17:38:05.827 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2023-07-14T17:38:05.827 INFO:teuthology.orchestra.run.smithi107.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111635 s, 939 MB/s 2023-07-14T17:38:05.827 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 2023-07-14T17:38:05.827 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2023-07-14T17:38:06.258 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi107:vg_nvme/lv_4 2023-07-14T17:38:06.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:06 smithi178 ceph-mon[129368]: from='osd.0 [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T17:38:06.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:06 smithi178 ceph-mon[129368]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T17:38:06.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:06 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T17:38:06.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:06 smithi178 ceph-mon[129368]: from='osd.0 [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T17:38:06.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:06 smithi178 ceph-mon[129368]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:38:06.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:06 smithi178 ceph-mon[129368]: Detected new or changed devices on smithi088 2023-07-14T17:38:06.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:06 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:06.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:06 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:06.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:06 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:38:06.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:06 smithi178 ceph-mon[129368]: Adjusting osd_memory_target on smithi088 to 17168M 2023-07-14T17:38:06.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:06 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:06.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:06 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:06.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:06 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:38:06.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:06 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:06.695 INFO:journalctl@ceph.osd.0.smithi088.stdout:Jul 14 17:38:06 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-0[140436]: 2023-07-14T17:38:06.340+0000 7fe363e83700 -1 osd.0 0 waiting for initial osdmap 2023-07-14T17:38:06.695 INFO:journalctl@ceph.osd.0.smithi088.stdout:Jul 14 17:38:06 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-0[140436]: 2023-07-14T17:38:06.343+0000 7fe35cc73700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T17:38:06.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:06 smithi088 ceph-mon[127465]: from='osd.0 [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T17:38:06.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:06 smithi088 ceph-mon[127465]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T17:38:06.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:06 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T17:38:06.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:06 smithi088 ceph-mon[127465]: from='osd.0 [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T17:38:06.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:06 smithi088 ceph-mon[127465]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:38:06.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:06 smithi088 ceph-mon[127465]: Detected new or changed devices on smithi088 2023-07-14T17:38:06.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:06 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:06.697 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:06 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:06.697 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:06 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:38:06.697 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:06 smithi088 ceph-mon[127465]: Adjusting osd_memory_target on smithi088 to 17168M 2023-07-14T17:38:06.697 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:06 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:06.697 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:06 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:06.697 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:06 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:38:06.697 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:06 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:06.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:06 smithi107 ceph-mon[128999]: from='osd.0 [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T17:38:06.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:06 smithi107 ceph-mon[128999]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T17:38:06.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:06 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T17:38:06.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:06 smithi107 ceph-mon[128999]: from='osd.0 [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T17:38:06.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:06 smithi107 ceph-mon[128999]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:38:06.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:06 smithi107 ceph-mon[128999]: Detected new or changed devices on smithi088 2023-07-14T17:38:06.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:06 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:06.730 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:06 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:06.730 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:06 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:38:06.730 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:06 smithi107 ceph-mon[128999]: Adjusting osd_memory_target on smithi088 to 17168M 2023-07-14T17:38:06.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:06 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:06.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:06 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:06.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:06 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:38:06.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:06 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:06.808 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T17:38:07.577 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:07 smithi107 ceph-mon[128999]: from='osd.0 [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T17:38:07.577 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:07 smithi107 ceph-mon[128999]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T17:38:07.577 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:07 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T17:38:07.577 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:07 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T17:38:07.577 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:07 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T17:38:07.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:07 smithi178 ceph-mon[129368]: from='osd.0 [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T17:38:07.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:07 smithi178 ceph-mon[129368]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T17:38:07.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:07 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T17:38:07.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:07 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T17:38:07.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:07 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T17:38:07.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:07 smithi088 ceph-mon[127465]: from='osd.0 [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T17:38:07.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:07 smithi088 ceph-mon[127465]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T17:38:07.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:07 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T17:38:07.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:07 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T17:38:07.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:07 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T17:38:08.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.035+0000 7f16e1386700 1 -- 172.21.15.107:0/1023895282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16dc1001a0 msgr2=0x7f16dc1005b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:38:08.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.035+0000 7f16e1386700 1 --2- 172.21.15.107:0/1023895282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16dc1001a0 0x7f16dc1005b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f16cc009990 tx=0x7f16cc009ca0 comp rx=0 tx=0).stop 2023-07-14T17:38:08.022 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.035+0000 7f16e1386700 1 -- 172.21.15.107:0/1023895282 shutdown_connections 2023-07-14T17:38:08.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.035+0000 7f16e1386700 1 --2- 172.21.15.107:0/1023895282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16dc0ff870 0x7f16dc107140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:08.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.035+0000 7f16e1386700 1 --2- 172.21.15.107:0/1023895282 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16dc1001a0 0x7f16dc1005b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:08.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.035+0000 7f16e1386700 1 --2- 172.21.15.107:0/1023895282 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16dc0feee0 0x7f16dc0ff330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:08.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.035+0000 7f16e1386700 1 -- 172.21.15.107:0/1023895282 >> 172.21.15.107:0/1023895282 conn(0x7f16dc0f99c0 msgr2=0x7f16dc0fbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:38:08.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.035+0000 7f16e1386700 1 -- 172.21.15.107:0/1023895282 shutdown_connections 2023-07-14T17:38:08.023 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.035+0000 7f16e1386700 1 -- 172.21.15.107:0/1023895282 wait complete. 2023-07-14T17:38:08.024 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.036+0000 7f16e1386700 1 Processor -- start 2023-07-14T17:38:08.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.036+0000 7f16e1386700 1 -- start start 2023-07-14T17:38:08.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.036+0000 7f16e1386700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16dc0feee0 0x7f16dc197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:38:08.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.036+0000 7f16e1386700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16dc0ff870 0x7f16dc198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:38:08.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.036+0000 7f16e1386700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16dc1001a0 0x7f16dc19f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:38:08.025 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.036+0000 7f16e1386700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc19f650 con 0x7f16dc1001a0 2023-07-14T17:38:08.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.036+0000 7f16e1386700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc19f790 con 0x7f16dc0feee0 2023-07-14T17:38:08.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.036+0000 7f16e1386700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16dc19f920 con 0x7f16dc0ff870 2023-07-14T17:38:08.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.036+0000 7f16daffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16dc0feee0 0x7f16dc197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:38:08.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.036+0000 7f16daffd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16dc0feee0 0x7f16dc197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42438/0 (socket says 172.21.15.107:42438) 2023-07-14T17:38:08.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.036+0000 7f16daffd700 1 -- 172.21.15.107:0/2491826543 learned_addr learned my addr 172.21.15.107:0/2491826543 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-07-14T17:38:08.026 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.036+0000 7f16db7fe700 1 --2- 172.21.15.107:0/2491826543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16dc1001a0 0x7f16dc19f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:38:08.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.037+0000 7f16da7fc700 1 --2- 172.21.15.107:0/2491826543 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16dc0ff870 0x7f16dc198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:38:08.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.037+0000 7f16daffd700 1 -- 172.21.15.107:0/2491826543 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16dc0ff870 msgr2=0x7f16dc198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:38:08.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.037+0000 7f16daffd700 1 --2- 172.21.15.107:0/2491826543 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16dc0ff870 0x7f16dc198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:08.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.037+0000 7f16daffd700 1 -- 172.21.15.107:0/2491826543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16dc1001a0 msgr2=0x7f16dc19f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:38:08.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.037+0000 7f16daffd700 1 --2- 172.21.15.107:0/2491826543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16dc1001a0 0x7f16dc19f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:08.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.037+0000 7f16daffd700 1 -- 172.21.15.107:0/2491826543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16cc0095f0 con 0x7f16dc0feee0 2023-07-14T17:38:08.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.037+0000 7f16daffd700 1 --2- 172.21.15.107:0/2491826543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16dc0feee0 0x7f16dc197ac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f16cc000c00 tx=0x7f16cc003a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:38:08.027 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.037+0000 7f16c3fff700 1 -- 172.21.15.107:0/2491826543 <== mon.2 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16cc01d070 con 0x7f16dc0feee0 2023-07-14T17:38:08.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.037+0000 7f16e1386700 1 -- 172.21.15.107:0/2491826543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16dc19fba0 con 0x7f16dc0feee0 2023-07-14T17:38:08.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.037+0000 7f16c3fff700 1 -- 172.21.15.107:0/2491826543 <== mon.2 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f16cc00bdc0 con 0x7f16dc0feee0 2023-07-14T17:38:08.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.037+0000 7f16c3fff700 1 -- 172.21.15.107:0/2491826543 <== mon.2 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16cc022e70 con 0x7f16dc0feee0 2023-07-14T17:38:08.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.037+0000 7f16e1386700 1 -- 172.21.15.107:0/2491826543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16dc1a0010 con 0x7f16dc0feee0 2023-07-14T17:38:08.028 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.038+0000 7f16e1386700 1 -- 172.21.15.107:0/2491826543 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16dc10a310 con 0x7f16dc0feee0 2023-07-14T17:38:08.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.039+0000 7f16c3fff700 1 -- 172.21.15.107:0/2491826543 <== mon.2 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f16cc003e80 con 0x7f16dc0feee0 2023-07-14T17:38:08.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.040+0000 7f16c3fff700 1 --2- 172.21.15.107:0/2491826543 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f16c40a10c0 0x7f16c40a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:38:08.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.040+0000 7f16c3fff700 1 -- 172.21.15.107:0/2491826543 <== mon.2 v2:172.21.15.107:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f16cc0cc5b0 con 0x7f16dc0feee0 2023-07-14T17:38:08.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.040+0000 7f16da7fc700 1 --2- 172.21.15.107:0/2491826543 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f16c40a10c0 0x7f16c40a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:38:08.029 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.041+0000 7f16da7fc700 1 --2- 172.21.15.107:0/2491826543 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f16c40a10c0 0x7f16c40a3570 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f16d0005ea0 tx=0x7f16d0005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:38:08.030 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.044+0000 7f16c3fff700 1 -- 172.21.15.107:0/2491826543 <== mon.2 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f16cc097800 con 0x7f16dc0feee0 2023-07-14T17:38:08.223 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:08.237+0000 7f16e1386700 1 -- 172.21.15.107:0/2491826543 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f16dc0fe580 con 0x7f16c40a10c0 2023-07-14T17:38:08.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:08 smithi178 ceph-mon[129368]: purged_snaps scrub starts 2023-07-14T17:38:08.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:08 smithi178 ceph-mon[129368]: purged_snaps scrub ok 2023-07-14T17:38:08.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:08 smithi178 ceph-mon[129368]: osd.0 [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341] boot 2023-07-14T17:38:08.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:08 smithi178 ceph-mon[129368]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T17:38:08.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:08 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T17:38:08.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:08 smithi178 ceph-mon[129368]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:38:08.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:08 smithi178 ceph-mon[129368]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:38:08.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:08 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:38:08.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:08 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:38:08.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:08 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:08.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:08 smithi088 ceph-mon[127465]: purged_snaps scrub starts 2023-07-14T17:38:08.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:08 smithi088 ceph-mon[127465]: purged_snaps scrub ok 2023-07-14T17:38:08.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:08 smithi088 ceph-mon[127465]: osd.0 [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341] boot 2023-07-14T17:38:08.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:08 smithi088 ceph-mon[127465]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T17:38:08.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:08 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T17:38:08.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:08 smithi088 ceph-mon[127465]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:38:08.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:08 smithi088 ceph-mon[127465]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:38:08.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:08 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:38:08.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:08 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:38:08.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:08 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:08.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:08 smithi107 ceph-mon[128999]: purged_snaps scrub starts 2023-07-14T17:38:08.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:08 smithi107 ceph-mon[128999]: purged_snaps scrub ok 2023-07-14T17:38:08.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:08 smithi107 ceph-mon[128999]: osd.0 [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341] boot 2023-07-14T17:38:08.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:08 smithi107 ceph-mon[128999]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T17:38:08.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:08 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T17:38:08.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:08 smithi107 ceph-mon[128999]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T17:38:08.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:08 smithi107 ceph-mon[128999]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:38:08.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:08 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:38:08.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:08 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:38:08.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:08 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:09.584 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:09 smithi107 ceph-mon[128999]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T17:38:09.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:09 smithi178 ceph-mon[129368]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T17:38:09.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:09 smithi088 ceph-mon[127465]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T17:38:10.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:10 smithi178 ceph-mon[129368]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:10.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:10 smithi088 ceph-mon[127465]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:10.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:10 smithi107 ceph-mon[128999]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:11.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:11 smithi178 ceph-mon[129368]: from='client.? 172.21.15.107:0/4108687798' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d11240d-b997-40ba-b67b-f39d92dce9b9"}]: dispatch 2023-07-14T17:38:11.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:11 smithi178 ceph-mon[129368]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d11240d-b997-40ba-b67b-f39d92dce9b9"}]: dispatch 2023-07-14T17:38:11.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:11 smithi178 ceph-mon[129368]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d11240d-b997-40ba-b67b-f39d92dce9b9"}]': finished 2023-07-14T17:38:11.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:11 smithi178 ceph-mon[129368]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T17:38:11.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:11 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T17:38:11.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:11 smithi178 ceph-mon[129368]: from='client.? 172.21.15.107:0/4027783376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:38:11.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:11 smithi088 ceph-mon[127465]: from='client.? 172.21.15.107:0/4108687798' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d11240d-b997-40ba-b67b-f39d92dce9b9"}]: dispatch 2023-07-14T17:38:11.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:11 smithi088 ceph-mon[127465]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d11240d-b997-40ba-b67b-f39d92dce9b9"}]: dispatch 2023-07-14T17:38:11.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:11 smithi088 ceph-mon[127465]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d11240d-b997-40ba-b67b-f39d92dce9b9"}]': finished 2023-07-14T17:38:11.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:11 smithi088 ceph-mon[127465]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T17:38:11.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:11 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T17:38:11.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:11 smithi088 ceph-mon[127465]: from='client.? 172.21.15.107:0/4027783376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:38:11.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:11 smithi107 ceph-mon[128999]: from='client.? 172.21.15.107:0/4108687798' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d11240d-b997-40ba-b67b-f39d92dce9b9"}]: dispatch 2023-07-14T17:38:11.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:11 smithi107 ceph-mon[128999]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d11240d-b997-40ba-b67b-f39d92dce9b9"}]: dispatch 2023-07-14T17:38:11.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:11 smithi107 ceph-mon[128999]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d11240d-b997-40ba-b67b-f39d92dce9b9"}]': finished 2023-07-14T17:38:11.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:11 smithi107 ceph-mon[128999]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T17:38:11.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:11 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T17:38:11.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:11 smithi107 ceph-mon[128999]: from='client.? 172.21.15.107:0/4027783376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:38:12.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:12 smithi178 ceph-mon[129368]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:12.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:12 smithi088 ceph-mon[127465]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:12.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:12 smithi107 ceph-mon[128999]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:15.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:14 smithi178 ceph-mon[129368]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:15.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:14 smithi088 ceph-mon[127465]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:15.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:14 smithi107 ceph-mon[128999]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:16.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:16 smithi178 ceph-mon[129368]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:16.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:16 smithi088 ceph-mon[127465]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:16.952 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:16 smithi107 ceph-mon[128999]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:17.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:17 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T17:38:17.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:17 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:17.920 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:17 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T17:38:17.920 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:17 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:17.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:17 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T17:38:17.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:17 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:18.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:18 smithi178 ceph-mon[129368]: Deploying daemon osd.1 on smithi107 2023-07-14T17:38:18.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:18 smithi178 ceph-mon[129368]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:18.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:18 smithi088 ceph-mon[127465]: Deploying daemon osd.1 on smithi107 2023-07-14T17:38:18.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:18 smithi088 ceph-mon[127465]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:18.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:18 smithi107 ceph-mon[128999]: Deploying daemon osd.1 on smithi107 2023-07-14T17:38:18.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:18 smithi107 ceph-mon[128999]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:21.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:20 smithi178 ceph-mon[129368]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:21.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:20 smithi088 ceph-mon[127465]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:21.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:20 smithi107 ceph-mon[128999]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:22.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:22 smithi107 ceph-mon[128999]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:22.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:22 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:22.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:22 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:22.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:22 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:38:23.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:22 smithi178 ceph-mon[129368]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:23.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:22 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:23.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:22 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:23.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:22 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:38:23.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:22 smithi088 ceph-mon[127465]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:23.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:22 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:23.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:22 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:23.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:22 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:38:24.736 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:24.750+0000 7f16c3fff700 1 -- 172.21.15.107:0/2491826543 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f16dc0fe580 con 0x7f16c40a10c0 2023-07-14T17:38:24.737 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 1 on host 'smithi107' 2023-07-14T17:38:24.738 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:24.753+0000 7f16e1386700 1 -- 172.21.15.107:0/2491826543 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f16c40a10c0 msgr2=0x7f16c40a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:38:24.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:24.753+0000 7f16e1386700 1 --2- 172.21.15.107:0/2491826543 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f16c40a10c0 0x7f16c40a3570 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f16d0005ea0 tx=0x7f16d0005e30 comp rx=0 tx=0).stop 2023-07-14T17:38:24.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:24.753+0000 7f16e1386700 1 -- 172.21.15.107:0/2491826543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16dc0feee0 msgr2=0x7f16dc197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:38:24.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:24.753+0000 7f16e1386700 1 --2- 172.21.15.107:0/2491826543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16dc0feee0 0x7f16dc197ac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f16cc000c00 tx=0x7f16cc003a50 comp rx=0 tx=0).stop 2023-07-14T17:38:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:24.753+0000 7f16e1386700 1 -- 172.21.15.107:0/2491826543 shutdown_connections 2023-07-14T17:38:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:24.753+0000 7f16e1386700 1 --2- 172.21.15.107:0/2491826543 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f16c40a10c0 0x7f16c40a3570 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:24.753+0000 7f16e1386700 1 --2- 172.21.15.107:0/2491826543 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16dc1001a0 0x7f16dc19f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:24.753+0000 7f16e1386700 1 --2- 172.21.15.107:0/2491826543 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16dc0feee0 0x7f16dc197ac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:24.753+0000 7f16e1386700 1 --2- 172.21.15.107:0/2491826543 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f16dc0ff870 0x7f16dc198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:24.753+0000 7f16e1386700 1 -- 172.21.15.107:0/2491826543 >> 172.21.15.107:0/2491826543 conn(0x7f16dc0f99c0 msgr2=0x7f16dc0fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:38:24.740 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:24.753+0000 7f16e1386700 1 -- 172.21.15.107:0/2491826543 shutdown_connections 2023-07-14T17:38:24.741 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:38:24.753+0000 7f16e1386700 1 -- 172.21.15.107:0/2491826543 wait complete. 2023-07-14T17:38:25.197 DEBUG:teuthology.orchestra.run.smithi107:osd.1> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.1.service 2023-07-14T17:38:25.199 INFO:tasks.cephadm:Deploying osd.2 on smithi178 with /dev/vg_nvme/lv_4... 2023-07-14T17:38:25.199 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T17:38:25.231 INFO:journalctl@ceph.osd.1.smithi107.stdout:-- Logs begin at Fri 2023-07-14 17:14:00 UTC. -- 2023-07-14T17:38:25.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:25 smithi178 ceph-mon[129368]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:25.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:25.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:25.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:25.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:38:25.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:25.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:25.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:25.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:38:25.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:25 smithi088 ceph-mon[127465]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:25.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:25.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:25.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:25.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:38:25.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:25.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:25.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:25.446 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:38:25.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:25 smithi107 ceph-mon[128999]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:25.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:25 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:25.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:25 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:25.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:25 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:25.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:25 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:38:25.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:25 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:25.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:25 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:25.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:25 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:25.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:25 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:38:25.759 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T17:38:25.977 INFO:journalctl@ceph.osd.1.smithi107.stdout:Jul 14 17:38:25 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-1[133955]: 2023-07-14T17:38:25.831+0000 7f571eab2640 -1 osd.1 0 log_to_monitors true 2023-07-14T17:38:26.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:26 smithi107 ceph-mon[128999]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:26.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:26 smithi107 ceph-mon[128999]: from='osd.1 [v2:172.21.15.107:6800/1795455333,v1:172.21.15.107:6801/1795455333]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T17:38:26.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:26 smithi107 ceph-mon[128999]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T17:38:26.875 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:26 smithi178 ceph-mon[129368]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:26.875 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:26 smithi178 ceph-mon[129368]: from='osd.1 [v2:172.21.15.107:6800/1795455333,v1:172.21.15.107:6801/1795455333]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T17:38:26.875 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:26 smithi178 ceph-mon[129368]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T17:38:26.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:26 smithi088 ceph-mon[127465]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:26.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:26 smithi088 ceph-mon[127465]: from='osd.1 [v2:172.21.15.107:6800/1795455333,v1:172.21.15.107:6801/1795455333]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T17:38:26.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:26 smithi088 ceph-mon[127465]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T17:38:27.336 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T17:38:27.336 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T17:38:27.336 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2023-07-14T17:38:27.337 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2023-07-14T17:38:27.337 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122298 s, 857 MB/s 2023-07-14T17:38:27.337 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2023-07-14T17:38:27.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:27 smithi178 ceph-mon[129368]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T17:38:27.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:27 smithi178 ceph-mon[129368]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T17:38:27.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:27 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T17:38:27.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:27 smithi178 ceph-mon[129368]: from='osd.1 [v2:172.21.15.107:6800/1795455333,v1:172.21.15.107:6801/1795455333]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:38:27.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:27 smithi178 ceph-mon[129368]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:38:27.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:27 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:27.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:27 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:27.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:27 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:38:27.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:27 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:27.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:27 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:27.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:27 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:38:27.669 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:27 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:27.719 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi178:vg_nvme/lv_4 2023-07-14T17:38:27.727 INFO:journalctl@ceph.osd.1.smithi107.stdout:Jul 14 17:38:27 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-1[133955]: 2023-07-14T17:38:27.643+0000 7f571412a700 -1 osd.1 0 waiting for initial osdmap 2023-07-14T17:38:27.727 INFO:journalctl@ceph.osd.1.smithi107.stdout:Jul 14 17:38:27 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-1[133955]: 2023-07-14T17:38:27.646+0000 7f570bf18700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T17:38:27.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:27 smithi107 ceph-mon[128999]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T17:38:27.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:27 smithi107 ceph-mon[128999]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T17:38:27.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:27 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T17:38:27.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:27 smithi107 ceph-mon[128999]: from='osd.1 [v2:172.21.15.107:6800/1795455333,v1:172.21.15.107:6801/1795455333]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:38:27.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:27 smithi107 ceph-mon[128999]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:38:27.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:27 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:27.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:27 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:27.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:27 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:38:27.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:27 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:27.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:27 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:27.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:27 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:38:27.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:27 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:27.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:27 smithi088 ceph-mon[127465]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T17:38:27.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:27 smithi088 ceph-mon[127465]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T17:38:27.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:27 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T17:38:27.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:27 smithi088 ceph-mon[127465]: from='osd.1 [v2:172.21.15.107:6800/1795455333,v1:172.21.15.107:6801/1795455333]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:38:27.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:27 smithi088 ceph-mon[127465]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:38:27.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:27 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:27.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:27 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:27.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:27 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:38:27.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:27 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:27.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:27 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:27.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:27 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:38:27.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:27 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:28.268 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T17:38:28.803 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:28 smithi178 ceph-mon[129368]: Detected new or changed devices on smithi107 2023-07-14T17:38:28.804 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:28 smithi178 ceph-mon[129368]: Adjusting osd_memory_target on smithi107 to 17168M 2023-07-14T17:38:28.804 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:28 smithi178 ceph-mon[129368]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2023-07-14T17:38:28.804 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:28 smithi178 ceph-mon[129368]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T17:38:28.804 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:28 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T17:38:28.804 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:28 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T17:38:28.805 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:28 smithi178 ceph-mon[129368]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:28.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:28 smithi088 ceph-mon[127465]: Detected new or changed devices on smithi107 2023-07-14T17:38:28.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:28 smithi088 ceph-mon[127465]: Adjusting osd_memory_target on smithi107 to 17168M 2023-07-14T17:38:28.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:28 smithi088 ceph-mon[127465]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2023-07-14T17:38:28.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:28 smithi088 ceph-mon[127465]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T17:38:28.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:28 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T17:38:28.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:28 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T17:38:28.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:28 smithi088 ceph-mon[127465]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:28.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:28 smithi107 ceph-mon[128999]: Detected new or changed devices on smithi107 2023-07-14T17:38:28.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:28 smithi107 ceph-mon[128999]: Adjusting osd_memory_target on smithi107 to 17168M 2023-07-14T17:38:28.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:28 smithi107 ceph-mon[128999]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2023-07-14T17:38:28.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:28 smithi107 ceph-mon[128999]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T17:38:28.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T17:38:28.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T17:38:28.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:28 smithi107 ceph-mon[128999]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T17:38:29.450 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.465+0000 7fcf8d82b700 1 -- 172.21.15.178:0/2228414873 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf880fdfd0 msgr2=0x7fcf880fe3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:38:29.451 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.465+0000 7fcf8d82b700 1 --2- 172.21.15.178:0/2228414873 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf880fdfd0 0x7fcf880fe3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcf78009990 tx=0x7fcf78009ca0 comp rx=0 tx=0).stop 2023-07-14T17:38:29.451 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.465+0000 7fcf8d82b700 1 -- 172.21.15.178:0/2228414873 shutdown_connections 2023-07-14T17:38:29.451 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.465+0000 7fcf8d82b700 1 --2- 172.21.15.178:0/2228414873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf881019e0 0x7fcf881082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:29.451 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.465+0000 7fcf8d82b700 1 --2- 172.21.15.178:0/2228414873 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf880fdfd0 0x7fcf880fe3e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:29.452 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.465+0000 7fcf8d82b700 1 --2- 172.21.15.178:0/2228414873 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf88101050 0x7fcf881014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:29.452 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.465+0000 7fcf8d82b700 1 -- 172.21.15.178:0/2228414873 >> 172.21.15.178:0/2228414873 conn(0x7fcf880f99c0 msgr2=0x7fcf880fbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:38:29.452 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.465+0000 7fcf8d82b700 1 -- 172.21.15.178:0/2228414873 shutdown_connections 2023-07-14T17:38:29.452 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.465+0000 7fcf8d82b700 1 -- 172.21.15.178:0/2228414873 wait complete. 2023-07-14T17:38:29.453 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.466+0000 7fcf8d82b700 1 Processor -- start 2023-07-14T17:38:29.454 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.466+0000 7fcf8d82b700 1 -- start start 2023-07-14T17:38:29.454 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.466+0000 7fcf8d82b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf880fdfd0 0x7fcf88195870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:38:29.454 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.466+0000 7fcf8d82b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf88101050 0x7fcf88195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:38:29.454 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.466+0000 7fcf8d82b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf881019e0 0x7fcf8819ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:38:29.454 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.466+0000 7fcf8d82b700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8819d400 con 0x7fcf881019e0 2023-07-14T17:38:29.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.466+0000 7fcf8d82b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8819d540 con 0x7fcf88101050 2023-07-14T17:38:29.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.466+0000 7fcf8d82b700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8819d6d0 con 0x7fcf880fdfd0 2023-07-14T17:38:29.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.466+0000 7fcf86ffd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf880fdfd0 0x7fcf88195870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:38:29.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.466+0000 7fcf86ffd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf880fdfd0 0x7fcf88195870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:54648/0 (socket says 172.21.15.178:54648) 2023-07-14T17:38:29.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.466+0000 7fcf86ffd700 1 -- 172.21.15.178:0/3261801199 learned_addr learned my addr 172.21.15.178:0/3261801199 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-07-14T17:38:29.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.466+0000 7fcf867fc700 1 --2- 172.21.15.178:0/3261801199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf88101050 0x7fcf88195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:38:29.455 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.467+0000 7fcf877fe700 1 --2- 172.21.15.178:0/3261801199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf881019e0 0x7fcf8819ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:38:29.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.467+0000 7fcf86ffd700 1 -- 172.21.15.178:0/3261801199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf88101050 msgr2=0x7fcf88195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:38:29.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.467+0000 7fcf86ffd700 1 --2- 172.21.15.178:0/3261801199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf88101050 0x7fcf88195db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:29.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.467+0000 7fcf86ffd700 1 -- 172.21.15.178:0/3261801199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf881019e0 msgr2=0x7fcf8819ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:38:29.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.467+0000 7fcf86ffd700 1 --2- 172.21.15.178:0/3261801199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf881019e0 0x7fcf8819ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:29.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.467+0000 7fcf86ffd700 1 -- 172.21.15.178:0/3261801199 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf780095f0 con 0x7fcf880fdfd0 2023-07-14T17:38:29.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.467+0000 7fcf86ffd700 1 --2- 172.21.15.178:0/3261801199 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf880fdfd0 0x7fcf88195870 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcf78004730 tx=0x7fcf78005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:38:29.456 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.467+0000 7fcf8c829700 1 -- 172.21.15.178:0/3261801199 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7801d070 con 0x7fcf880fdfd0 2023-07-14T17:38:29.457 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.467+0000 7fcf8d82b700 1 -- 172.21.15.178:0/3261801199 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf8819d950 con 0x7fcf880fdfd0 2023-07-14T17:38:29.457 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.467+0000 7fcf8c829700 1 -- 172.21.15.178:0/3261801199 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcf78004100 con 0x7fcf880fdfd0 2023-07-14T17:38:29.457 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.467+0000 7fcf8c829700 1 -- 172.21.15.178:0/3261801199 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf78005290 con 0x7fcf880fdfd0 2023-07-14T17:38:29.457 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.467+0000 7fcf8d82b700 1 -- 172.21.15.178:0/3261801199 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf8819dd70 con 0x7fcf880fdfd0 2023-07-14T17:38:29.457 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.468+0000 7fcf8d82b700 1 -- 172.21.15.178:0/3261801199 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf881073d0 con 0x7fcf880fdfd0 2023-07-14T17:38:29.458 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.470+0000 7fcf8c829700 1 -- 172.21.15.178:0/3261801199 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7fcf78004810 con 0x7fcf880fdfd0 2023-07-14T17:38:29.458 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.470+0000 7fcf8c829700 1 --2- 172.21.15.178:0/3261801199 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcf700a10c0 0x7fcf700a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:38:29.458 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.470+0000 7fcf8c829700 1 -- 172.21.15.178:0/3261801199 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2309+0+0 (secure 0 0 0) 0x7fcf780cc510 con 0x7fcf880fdfd0 2023-07-14T17:38:29.458 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.470+0000 7fcf867fc700 1 --2- 172.21.15.178:0/3261801199 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcf700a10c0 0x7fcf700a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:38:29.458 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.471+0000 7fcf867fc700 1 --2- 172.21.15.178:0/3261801199 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcf700a10c0 0x7fcf700a3570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c005950 tx=0x7fcf7c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:38:29.459 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.474+0000 7fcf8c829700 1 -- 172.21.15.178:0/3261801199 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcf78097540 con 0x7fcf880fdfd0 2023-07-14T17:38:29.652 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:29.667+0000 7fcf8d82b700 1 -- 172.21.15.178:0/3261801199 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf88060a80 con 0x7fcf700a10c0 2023-07-14T17:38:29.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:29 smithi178 ceph-mon[129368]: purged_snaps scrub starts 2023-07-14T17:38:29.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:29 smithi178 ceph-mon[129368]: purged_snaps scrub ok 2023-07-14T17:38:29.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:29 smithi178 ceph-mon[129368]: osd.1 [v2:172.21.15.107:6800/1795455333,v1:172.21.15.107:6801/1795455333] boot 2023-07-14T17:38:29.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:29 smithi178 ceph-mon[129368]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T17:38:29.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:29 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T17:38:29.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:29 smithi088 ceph-mon[127465]: purged_snaps scrub starts 2023-07-14T17:38:29.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:29 smithi088 ceph-mon[127465]: purged_snaps scrub ok 2023-07-14T17:38:29.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:29 smithi088 ceph-mon[127465]: osd.1 [v2:172.21.15.107:6800/1795455333,v1:172.21.15.107:6801/1795455333] boot 2023-07-14T17:38:29.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:29 smithi088 ceph-mon[127465]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T17:38:29.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:29 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T17:38:29.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:29 smithi107 ceph-mon[128999]: purged_snaps scrub starts 2023-07-14T17:38:29.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:29 smithi107 ceph-mon[128999]: purged_snaps scrub ok 2023-07-14T17:38:29.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:29 smithi107 ceph-mon[128999]: osd.1 [v2:172.21.15.107:6800/1795455333,v1:172.21.15.107:6801/1795455333] boot 2023-07-14T17:38:29.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:29 smithi107 ceph-mon[128999]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T17:38:29.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:29 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T17:38:30.789 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:30 smithi178 ceph-mon[129368]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T17:38:30.789 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:30 smithi178 ceph-mon[129368]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:38:30.789 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:30 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:38:30.789 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:30 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:38:30.789 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:30 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:30.790 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:30 smithi178 ceph-mon[129368]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:30.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:30 smithi088 ceph-mon[127465]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T17:38:30.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:30 smithi088 ceph-mon[127465]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:38:30.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:30 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:38:30.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:30 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:38:30.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:30 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:30.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:30 smithi088 ceph-mon[127465]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:30.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:30 smithi107 ceph-mon[128999]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T17:38:30.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:30 smithi107 ceph-mon[128999]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:38:30.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:30 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:38:30.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:30 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:38:30.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:30 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:30.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:30 smithi107 ceph-mon[128999]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:33.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:32 smithi178 ceph-mon[129368]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:33.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:32 smithi178 ceph-mon[129368]: from='client.? 172.21.15.178:0/773768125' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41542091-8564-496c-9b43-a827f60ac844"}]: dispatch 2023-07-14T17:38:33.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:32 smithi178 ceph-mon[129368]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41542091-8564-496c-9b43-a827f60ac844"}]: dispatch 2023-07-14T17:38:33.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:32 smithi178 ceph-mon[129368]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41542091-8564-496c-9b43-a827f60ac844"}]': finished 2023-07-14T17:38:33.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:32 smithi178 ceph-mon[129368]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T17:38:33.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T17:38:33.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:32 smithi178 ceph-mon[129368]: from='client.? 172.21.15.178:0/3828829247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:38:33.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:32 smithi088 ceph-mon[127465]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:33.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:32 smithi088 ceph-mon[127465]: from='client.? 172.21.15.178:0/773768125' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41542091-8564-496c-9b43-a827f60ac844"}]: dispatch 2023-07-14T17:38:33.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:32 smithi088 ceph-mon[127465]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41542091-8564-496c-9b43-a827f60ac844"}]: dispatch 2023-07-14T17:38:33.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:32 smithi088 ceph-mon[127465]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41542091-8564-496c-9b43-a827f60ac844"}]': finished 2023-07-14T17:38:33.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:32 smithi088 ceph-mon[127465]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T17:38:33.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T17:38:33.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:32 smithi088 ceph-mon[127465]: from='client.? 172.21.15.178:0/3828829247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:38:33.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:32 smithi107 ceph-mon[128999]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:33.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:32 smithi107 ceph-mon[128999]: from='client.? 172.21.15.178:0/773768125' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41542091-8564-496c-9b43-a827f60ac844"}]: dispatch 2023-07-14T17:38:33.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:32 smithi107 ceph-mon[128999]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "41542091-8564-496c-9b43-a827f60ac844"}]: dispatch 2023-07-14T17:38:33.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:32 smithi107 ceph-mon[128999]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "41542091-8564-496c-9b43-a827f60ac844"}]': finished 2023-07-14T17:38:33.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:32 smithi107 ceph-mon[128999]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T17:38:33.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T17:38:33.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:32 smithi107 ceph-mon[128999]: from='client.? 172.21.15.178:0/3828829247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:38:35.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:34 smithi178 ceph-mon[129368]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:35.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:34 smithi088 ceph-mon[127465]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:35.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:34 smithi107 ceph-mon[128999]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:36.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:36 smithi178 ceph-mon[129368]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:36.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:36 smithi088 ceph-mon[127465]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:36.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:36 smithi107 ceph-mon[128999]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:39.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:38 smithi178 ceph-mon[129368]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:39.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:38 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T17:38:39.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:38 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:39.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:38 smithi088 ceph-mon[127465]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:39.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:38 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T17:38:39.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:38 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:39.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:38 smithi107 ceph-mon[128999]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:39.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:38 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T17:38:39.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:38 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:40.078 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:39 smithi178 ceph-mon[129368]: Deploying daemon osd.2 on smithi178 2023-07-14T17:38:40.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:39 smithi088 ceph-mon[127465]: Deploying daemon osd.2 on smithi178 2023-07-14T17:38:40.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:39 smithi107 ceph-mon[128999]: Deploying daemon osd.2 on smithi178 2023-07-14T17:38:41.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:40 smithi178 ceph-mon[129368]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:41.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:40 smithi088 ceph-mon[127465]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:41.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:40 smithi107 ceph-mon[128999]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:43.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:42 smithi178 ceph-mon[129368]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:43.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:42 smithi088 ceph-mon[127465]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:43.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:42 smithi107 ceph-mon[128999]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:44.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:44.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:44.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:38:44.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:44 smithi178 ceph-mon[129368]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:44.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:44.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:44.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:38:44.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:44 smithi088 ceph-mon[127465]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:44.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:44.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:44.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:38:44.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:44 smithi107 ceph-mon[128999]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:46.362 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 2 on host 'smithi178' 2023-07-14T17:38:46.362 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:46.376+0000 7fcf8c829700 1 -- 172.21.15.178:0/3261801199 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcf88060a80 con 0x7fcf700a10c0 2023-07-14T17:38:46.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:46.378+0000 7fcf8d82b700 1 -- 172.21.15.178:0/3261801199 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcf700a10c0 msgr2=0x7fcf700a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:38:46.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:46.378+0000 7fcf8d82b700 1 --2- 172.21.15.178:0/3261801199 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcf700a10c0 0x7fcf700a3570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c005950 tx=0x7fcf7c0058e0 comp rx=0 tx=0).stop 2023-07-14T17:38:46.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:46.378+0000 7fcf8d82b700 1 -- 172.21.15.178:0/3261801199 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf880fdfd0 msgr2=0x7fcf88195870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:38:46.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:46.378+0000 7fcf8d82b700 1 --2- 172.21.15.178:0/3261801199 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf880fdfd0 0x7fcf88195870 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcf78004730 tx=0x7fcf78005ef0 comp rx=0 tx=0).stop 2023-07-14T17:38:46.364 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:46.378+0000 7fcf8d82b700 1 -- 172.21.15.178:0/3261801199 shutdown_connections 2023-07-14T17:38:46.365 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:46.378+0000 7fcf8d82b700 1 --2- 172.21.15.178:0/3261801199 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcf700a10c0 0x7fcf700a3570 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:46.365 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:46.378+0000 7fcf8d82b700 1 --2- 172.21.15.178:0/3261801199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf881019e0 0x7fcf8819ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:46.365 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:46.378+0000 7fcf8d82b700 1 --2- 172.21.15.178:0/3261801199 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcf880fdfd0 0x7fcf88195870 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:46.365 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:46.378+0000 7fcf8d82b700 1 --2- 172.21.15.178:0/3261801199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcf88101050 0x7fcf88195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:46.365 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:46.378+0000 7fcf8d82b700 1 -- 172.21.15.178:0/3261801199 >> 172.21.15.178:0/3261801199 conn(0x7fcf880f99c0 msgr2=0x7fcf88104510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:38:46.365 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:46.378+0000 7fcf8d82b700 1 -- 172.21.15.178:0/3261801199 shutdown_connections 2023-07-14T17:38:46.365 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:38:46.378+0000 7fcf8d82b700 1 -- 172.21.15.178:0/3261801199 wait complete. 2023-07-14T17:38:46.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:46 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:46.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:46 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:46.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:46 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:46.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:46 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:38:46.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:46 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:46.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:46 smithi178 ceph-mon[129368]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:46.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:46 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:46.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:46 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:46.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:46 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:38:46.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:46 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:46.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:46 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:46.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:46 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:46.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:46 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:38:46.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:46 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:46.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:46 smithi107 ceph-mon[128999]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:46.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:46 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:46.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:46 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:46.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:46 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:38:46.747 DEBUG:teuthology.orchestra.run.smithi178:osd.2> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.2.service 2023-07-14T17:38:46.749 INFO:tasks.cephadm:Deploying osd.3 on smithi088 with /dev/vg_nvme/lv_3... 2023-07-14T17:38:46.749 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T17:38:46.777 INFO:journalctl@ceph.osd.2.smithi178.stdout:-- Logs begin at Fri 2023-07-14 17:14:18 UTC. -- 2023-07-14T17:38:46.781 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:46 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:46.782 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:46 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:46.782 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:46 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:46.782 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:46 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:38:46.782 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:46 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:46.782 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:46 smithi088 ceph-mon[127465]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:46.783 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:46 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:46.783 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:46 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:46.783 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:46 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:38:47.317 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:38:47.622 INFO:journalctl@ceph.osd.2.smithi178.stdout:Jul 14 17:38:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-2[135188]: 2023-07-14T17:38:47.385+0000 7fd1ce18d640 -1 osd.2 0 log_to_monitors true 2023-07-14T17:38:47.906 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:47 smithi088 ceph-mon[127465]: from='osd.2 [v2:172.21.15.178:6800/4021325462,v1:172.21.15.178:6801/4021325462]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T17:38:47.907 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:47 smithi088 ceph-mon[127465]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T17:38:47.912 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:47 smithi178 ceph-mon[129368]: from='osd.2 [v2:172.21.15.178:6800/4021325462,v1:172.21.15.178:6801/4021325462]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T17:38:47.912 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:47 smithi178 ceph-mon[129368]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T17:38:47.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:47 smithi107 ceph-mon[128999]: from='osd.2 [v2:172.21.15.178:6800/4021325462,v1:172.21.15.178:6801/4021325462]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T17:38:47.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:47 smithi107 ceph-mon[128999]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T17:38:48.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:48 smithi088 ceph-mon[127465]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T17:38:48.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:48 smithi088 ceph-mon[127465]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T17:38:48.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:48 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T17:38:48.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:48 smithi088 ceph-mon[127465]: from='osd.2 [v2:172.21.15.178:6800/4021325462,v1:172.21.15.178:6801/4021325462]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:38:48.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:48 smithi088 ceph-mon[127465]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:38:48.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:48 smithi088 ceph-mon[127465]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:48.902 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:48 smithi178 ceph-mon[129368]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T17:38:48.903 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:48 smithi178 ceph-mon[129368]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T17:38:48.903 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:48 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T17:38:48.903 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:48 smithi178 ceph-mon[129368]: from='osd.2 [v2:172.21.15.178:6800/4021325462,v1:172.21.15.178:6801/4021325462]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:38:48.903 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:48 smithi178 ceph-mon[129368]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:38:48.903 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:48 smithi178 ceph-mon[129368]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:48.903 INFO:journalctl@ceph.osd.2.smithi178.stdout:Jul 14 17:38:48 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-2[135188]: 2023-07-14T17:38:48.655+0000 7fd1c3805700 -1 osd.2 0 waiting for initial osdmap 2023-07-14T17:38:48.904 INFO:journalctl@ceph.osd.2.smithi178.stdout:Jul 14 17:38:48 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-2[135188]: 2023-07-14T17:38:48.658+0000 7fd1be5f9700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T17:38:48.904 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T17:38:48.904 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T17:38:48.904 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2023-07-14T17:38:48.904 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2023-07-14T17:38:48.905 INFO:teuthology.orchestra.run.smithi088.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0107356 s, 977 MB/s 2023-07-14T17:38:48.905 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2023-07-14T17:38:48.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:48 smithi107 ceph-mon[128999]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T17:38:48.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:48 smithi107 ceph-mon[128999]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T17:38:48.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:48 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T17:38:48.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:48 smithi107 ceph-mon[128999]: from='osd.2 [v2:172.21.15.178:6800/4021325462,v1:172.21.15.178:6801/4021325462]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:38:48.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:48 smithi107 ceph-mon[128999]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:38:48.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:48 smithi107 ceph-mon[128999]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:49.361 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi088:vg_nvme/lv_3 2023-07-14T17:38:49.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:49 smithi178 ceph-mon[129368]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2023-07-14T17:38:49.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:49 smithi178 ceph-mon[129368]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T17:38:49.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T17:38:49.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T17:38:49.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:49 smithi178 ceph-mon[129368]: Detected new or changed devices on smithi178 2023-07-14T17:38:49.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:49.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:49.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:38:49.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:49 smithi178 ceph-mon[129368]: Adjusting osd_memory_target on smithi178 to 17168M 2023-07-14T17:38:49.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:49.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:49.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:38:49.919 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:49.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:49 smithi088 ceph-mon[127465]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2023-07-14T17:38:49.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:49 smithi088 ceph-mon[127465]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T17:38:49.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T17:38:49.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T17:38:49.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:49 smithi088 ceph-mon[127465]: Detected new or changed devices on smithi178 2023-07-14T17:38:49.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:49.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:49.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:38:49.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:49 smithi088 ceph-mon[127465]: Adjusting osd_memory_target on smithi178 to 17168M 2023-07-14T17:38:49.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:49.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:49.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:38:49.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:49.948 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:38:49.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:49 smithi107 ceph-mon[128999]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2023-07-14T17:38:49.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:49 smithi107 ceph-mon[128999]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T17:38:49.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T17:38:49.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T17:38:49.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:49 smithi107 ceph-mon[128999]: Detected new or changed devices on smithi178 2023-07-14T17:38:49.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:49.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:49.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:38:49.979 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:49 smithi107 ceph-mon[128999]: Adjusting osd_memory_target on smithi178 to 17168M 2023-07-14T17:38:49.979 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:49.979 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:49.979 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:38:49.979 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:50.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:50 smithi178 ceph-mon[129368]: purged_snaps scrub starts 2023-07-14T17:38:50.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:50 smithi178 ceph-mon[129368]: purged_snaps scrub ok 2023-07-14T17:38:50.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:50 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T17:38:50.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:50 smithi178 ceph-mon[129368]: osd.2 [v2:172.21.15.178:6800/4021325462,v1:172.21.15.178:6801/4021325462] boot 2023-07-14T17:38:50.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:50 smithi178 ceph-mon[129368]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T17:38:50.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:50 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T17:38:50.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:50 smithi178 ceph-mon[129368]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:50.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:50 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T17:38:50.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:50 smithi088 ceph-mon[127465]: purged_snaps scrub starts 2023-07-14T17:38:50.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:50 smithi088 ceph-mon[127465]: purged_snaps scrub ok 2023-07-14T17:38:50.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:50 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T17:38:50.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:50 smithi088 ceph-mon[127465]: osd.2 [v2:172.21.15.178:6800/4021325462,v1:172.21.15.178:6801/4021325462] boot 2023-07-14T17:38:50.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:50 smithi088 ceph-mon[127465]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T17:38:50.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:50 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T17:38:50.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:50 smithi088 ceph-mon[127465]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:50.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:50 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T17:38:50.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:50 smithi107 ceph-mon[128999]: purged_snaps scrub starts 2023-07-14T17:38:50.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:50 smithi107 ceph-mon[128999]: purged_snaps scrub ok 2023-07-14T17:38:50.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:50 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T17:38:50.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:50 smithi107 ceph-mon[128999]: osd.2 [v2:172.21.15.178:6800/4021325462,v1:172.21.15.178:6801/4021325462] boot 2023-07-14T17:38:50.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:50 smithi107 ceph-mon[128999]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T17:38:50.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:50 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T17:38:50.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:50 smithi107 ceph-mon[128999]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T17:38:50.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:50 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T17:38:51.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.162+0000 7f786e185700 1 -- 172.21.15.88:0/2999218037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78680fe380 msgr2=0x7f78680fe790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:38:51.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.162+0000 7f786e185700 1 --2- 172.21.15.88:0/2999218037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78680fe380 0x7f78680fe790 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f785c009990 tx=0x7f785c009ca0 comp rx=0 tx=0).stop 2023-07-14T17:38:51.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.162+0000 7f786e185700 1 -- 172.21.15.88:0/2999218037 shutdown_connections 2023-07-14T17:38:51.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.162+0000 7f786e185700 1 --2- 172.21.15.88:0/2999218037 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78680fff10 0x7f78681067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:51.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.162+0000 7f786e185700 1 --2- 172.21.15.88:0/2999218037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78680fe380 0x7f78680fe790 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:51.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.162+0000 7f786e185700 1 --2- 172.21.15.88:0/2999218037 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78680ff580 0x7f78680ff9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:51.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.162+0000 7f786e185700 1 -- 172.21.15.88:0/2999218037 >> 172.21.15.88:0/2999218037 conn(0x7f78680f99b0 msgr2=0x7f78680fbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:38:51.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.162+0000 7f786e185700 1 -- 172.21.15.88:0/2999218037 shutdown_connections 2023-07-14T17:38:51.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.162+0000 7f786e185700 1 -- 172.21.15.88:0/2999218037 wait complete. 2023-07-14T17:38:51.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.162+0000 7f786e185700 1 Processor -- start 2023-07-14T17:38:51.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.163+0000 7f786e185700 1 -- start start 2023-07-14T17:38:51.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.163+0000 7f786e185700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78680fe380 0x7f78680fbe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:38:51.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.163+0000 7f786e185700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78680ff580 0x7f786819a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:38:51.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.163+0000 7f786e185700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78680fff10 0x7f786819cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:38:51.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.163+0000 7f786e185700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786819d4b0 con 0x7f78680fe380 2023-07-14T17:38:51.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.163+0000 7f786e185700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786819d620 con 0x7f78680ff580 2023-07-14T17:38:51.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.163+0000 7f786e185700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786819d7e0 con 0x7f78680fff10 2023-07-14T17:38:51.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.163+0000 7f78677fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78680fe380 0x7f78680fbe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:38:51.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.163+0000 7f78677fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78680fe380 0x7f78680fbe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40612/0 (socket says 172.21.15.88:40612) 2023-07-14T17:38:51.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.163+0000 7f78677fe700 1 -- 172.21.15.88:0/4109691264 learned_addr learned my addr 172.21.15.88:0/4109691264 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:38:51.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.163+0000 7f7867fff700 1 --2- 172.21.15.88:0/4109691264 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78680fff10 0x7f786819cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:38:51.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.163+0000 7f7866ffd700 1 --2- 172.21.15.88:0/4109691264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78680ff580 0x7f786819a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:38:51.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.163+0000 7f78677fe700 1 -- 172.21.15.88:0/4109691264 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78680fff10 msgr2=0x7f786819cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:38:51.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.163+0000 7f78677fe700 1 --2- 172.21.15.88:0/4109691264 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78680fff10 0x7f786819cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:51.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.163+0000 7f78677fe700 1 -- 172.21.15.88:0/4109691264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78680ff580 msgr2=0x7f786819a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:38:51.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.163+0000 7f78677fe700 1 --2- 172.21.15.88:0/4109691264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78680ff580 0x7f786819a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:38:51.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.163+0000 7f78677fe700 1 -- 172.21.15.88:0/4109691264 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785c0095f0 con 0x7f78680fe380 2023-07-14T17:38:51.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.163+0000 7f78677fe700 1 --2- 172.21.15.88:0/4109691264 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78680fe380 0x7f78680fbe60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f785c004730 tx=0x7f785c004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:38:51.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.164+0000 7f7864ff9700 1 -- 172.21.15.88:0/4109691264 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c01d070 con 0x7f78680fe380 2023-07-14T17:38:51.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.164+0000 7f7864ff9700 1 -- 172.21.15.88:0/4109691264 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f785c004100 con 0x7f78680fe380 2023-07-14T17:38:51.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.164+0000 7f786e185700 1 -- 172.21.15.88:0/4109691264 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f786819da60 con 0x7f78680fe380 2023-07-14T17:38:51.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.164+0000 7f7864ff9700 1 -- 172.21.15.88:0/4109691264 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c022dd0 con 0x7f78680fe380 2023-07-14T17:38:51.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.164+0000 7f786e185700 1 -- 172.21.15.88:0/4109691264 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f786819ded0 con 0x7f78680fe380 2023-07-14T17:38:51.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.164+0000 7f786e185700 1 -- 172.21.15.88:0/4109691264 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7868105f00 con 0x7f78680fe380 2023-07-14T17:38:51.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.166+0000 7f7864ff9700 1 -- 172.21.15.88:0/4109691264 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f785c003c70 con 0x7f78680fe380 2023-07-14T17:38:51.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.166+0000 7f7864ff9700 1 --2- 172.21.15.88:0/4109691264 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f78500a1140 0x7f78500a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:38:51.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.166+0000 7f7866ffd700 1 --2- 172.21.15.88:0/4109691264 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f78500a1140 0x7f78500a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:38:51.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.166+0000 7f7864ff9700 1 -- 172.21.15.88:0/4109691264 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 3226+0+0 (secure 0 0 0) 0x7f785c0cce90 con 0x7f78680fe380 2023-07-14T17:38:51.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.167+0000 7f7866ffd700 1 --2- 172.21.15.88:0/4109691264 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f78500a1140 0x7f78500a35f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78680fc3e0 tx=0x7f785800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:38:51.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.170+0000 7f7864ff9700 1 -- 172.21.15.88:0/4109691264 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f785c097b30 con 0x7f78680fe380 2023-07-14T17:38:51.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:51.363+0000 7f786e185700 1 -- 172.21.15.88:0/4109691264 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f7868060a80 con 0x7f78500a1140 2023-07-14T17:38:51.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:51 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T17:38:51.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:51 smithi178 ceph-mon[129368]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T17:38:51.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:51 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T17:38:51.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:51 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:38:51.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:51 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:38:51.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:51 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:51.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:51 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T17:38:51.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:51 smithi088 ceph-mon[127465]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T17:38:51.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:51 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T17:38:51.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:51 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:38:51.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:51 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:38:51.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:51 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:51.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:51 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T17:38:51.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:51 smithi107 ceph-mon[128999]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T17:38:51.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:51 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T17:38:51.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:51 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:38:51.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:51 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:38:51.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:51 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:38:52.259 INFO:journalctl@ceph.osd.0.smithi088.stdout:Jul 14 17:38:52 smithi088 sudo[143376]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T17:38:52.260 INFO:journalctl@ceph.osd.0.smithi088.stdout:Jul 14 17:38:52 smithi088 sudo[143376]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T17:38:52.260 INFO:journalctl@ceph.osd.0.smithi088.stdout:Jul 14 17:38:52 smithi088 sudo[143376]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T17:38:52.260 INFO:journalctl@ceph.osd.0.smithi088.stdout:Jul 14 17:38:52 smithi088 sudo[143376]: pam_unix(sudo:session): session closed for user root 2023-07-14T17:38:52.655 INFO:journalctl@ceph.osd.0.smithi088.stdout:Jul 14 17:38:52 smithi088 sudo[143400]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T17:38:52.655 INFO:journalctl@ceph.osd.0.smithi088.stdout:Jul 14 17:38:52 smithi088 sudo[143400]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T17:38:52.655 INFO:journalctl@ceph.osd.0.smithi088.stdout:Jul 14 17:38:52 smithi088 sudo[143400]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T17:38:52.656 INFO:journalctl@ceph.osd.0.smithi088.stdout:Jul 14 17:38:52 smithi088 sudo[143400]: pam_unix(sudo:session): session closed for user root 2023-07-14T17:38:52.782 INFO:journalctl@ceph.osd.1.smithi107.stdout:Jul 14 17:38:52 smithi107 sudo[136252]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T17:38:52.782 INFO:journalctl@ceph.osd.1.smithi107.stdout:Jul 14 17:38:52 smithi107 sudo[136252]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T17:38:52.782 INFO:journalctl@ceph.osd.1.smithi107.stdout:Jul 14 17:38:52 smithi107 sudo[136252]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T17:38:52.783 INFO:journalctl@ceph.osd.1.smithi107.stdout:Jul 14 17:38:52 smithi107 sudo[136252]: pam_unix(sudo:session): session closed for user root 2023-07-14T17:38:52.783 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:52 smithi107 ceph-mon[128999]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:38:52.783 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:52 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T17:38:52.783 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:52 smithi107 ceph-mon[128999]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T17:38:52.783 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:52 smithi107 ceph-mon[128999]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T17:38:52.783 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:52 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:52.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:52 smithi178 ceph-mon[129368]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:38:52.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:52 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T17:38:52.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:52 smithi178 ceph-mon[129368]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T17:38:52.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:52 smithi178 ceph-mon[129368]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T17:38:52.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:52 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:52.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:52 smithi088 ceph-mon[127465]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:38:52.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:52 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T17:38:52.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:52 smithi088 ceph-mon[127465]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T17:38:52.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:52 smithi088 ceph-mon[127465]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T17:38:52.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:52 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:53.227 INFO:journalctl@ceph.osd.1.smithi107.stdout:Jul 14 17:38:52 smithi107 sudo[136255]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T17:38:53.227 INFO:journalctl@ceph.osd.1.smithi107.stdout:Jul 14 17:38:52 smithi107 sudo[136255]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T17:38:53.227 INFO:journalctl@ceph.osd.1.smithi107.stdout:Jul 14 17:38:52 smithi107 sudo[136255]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T17:38:53.227 INFO:journalctl@ceph.osd.1.smithi107.stdout:Jul 14 17:38:52 smithi107 sudo[136255]: pam_unix(sudo:session): session closed for user root 2023-07-14T17:38:53.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:38:53.308+0000 7f7864ff9700 1 -- 172.21.15.88:0/4109691264 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f785c097f00 con 0x7f78680fe380 2023-07-14T17:38:53.304 INFO:journalctl@ceph.osd.2.smithi178.stdout:Jul 14 17:38:53 smithi178 sudo[137429]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T17:38:53.304 INFO:journalctl@ceph.osd.2.smithi178.stdout:Jul 14 17:38:53 smithi178 sudo[137429]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T17:38:53.304 INFO:journalctl@ceph.osd.2.smithi178.stdout:Jul 14 17:38:53 smithi178 sudo[137429]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T17:38:53.305 INFO:journalctl@ceph.osd.2.smithi178.stdout:Jul 14 17:38:53 smithi178 sudo[137429]: pam_unix(sudo:session): session closed for user root 2023-07-14T17:38:53.667 INFO:journalctl@ceph.osd.2.smithi178.stdout:Jul 14 17:38:53 smithi178 sudo[137432]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T17:38:53.667 INFO:journalctl@ceph.osd.2.smithi178.stdout:Jul 14 17:38:53 smithi178 sudo[137432]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T17:38:53.667 INFO:journalctl@ceph.osd.2.smithi178.stdout:Jul 14 17:38:53 smithi178 sudo[137432]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T17:38:53.667 INFO:journalctl@ceph.osd.2.smithi178.stdout:Jul 14 17:38:53 smithi178 sudo[137432]: pam_unix(sudo:session): session closed for user root 2023-07-14T17:38:53.847 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:53 smithi088 sudo[143507]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T17:38:53.847 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:53 smithi088 sudo[143507]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T17:38:53.848 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:53 smithi088 sudo[143507]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T17:38:53.848 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:53 smithi088 sudo[143507]: pam_unix(sudo:session): session closed for user root 2023-07-14T17:38:54.106 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:53 smithi178 ceph-mon[129368]: osdmap e21: 3 total, 3 up, 3 in 2023-07-14T17:38:54.106 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:53 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:54.106 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:53 smithi178 ceph-mon[129368]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T17:38:54.106 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:53 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:54.106 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:53 smithi178 ceph-mon[129368]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T17:38:54.107 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 sudo[137435]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T17:38:54.107 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 sudo[137435]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T17:38:54.107 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 sudo[137435]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T17:38:54.107 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 sudo[137435]: pam_unix(sudo:session): session closed for user root 2023-07-14T17:38:54.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:53 smithi088 sudo[143510]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T17:38:54.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:53 smithi088 sudo[143510]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T17:38:54.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:53 smithi088 sudo[143510]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T17:38:54.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:53 smithi088 sudo[143510]: pam_unix(sudo:session): session closed for user root 2023-07-14T17:38:54.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:53 smithi088 ceph-mon[127465]: osdmap e21: 3 total, 3 up, 3 in 2023-07-14T17:38:54.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:53 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:54.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:53 smithi088 ceph-mon[127465]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T17:38:54.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:53 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:54.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:53 smithi088 ceph-mon[127465]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T17:38:54.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:53 smithi107 ceph-mon[128999]: osdmap e21: 3 total, 3 up, 3 in 2023-07-14T17:38:54.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:53 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:54.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:53 smithi107 ceph-mon[128999]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T17:38:54.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:53 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:38:54.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:53 smithi107 ceph-mon[128999]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T17:38:54.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 sudo[137438]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T17:38:54.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 sudo[137438]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T17:38:54.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 sudo[137438]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T17:38:54.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 sudo[137438]: pam_unix(sudo:session): session closed for user root 2023-07-14T17:38:54.842 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 sudo[136258]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T17:38:54.842 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 sudo[136258]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T17:38:54.842 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 sudo[136258]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T17:38:54.842 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 sudo[136258]: pam_unix(sudo:session): session closed for user root 2023-07-14T17:38:55.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-14T17:38:55.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T17:38:55.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T17:38:55.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:38:55.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:38:55.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T17:38:55.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T17:38:55.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:38:55.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:38:55.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/4084923967' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77c095e1-8805-435b-bb0e-1ff3ef4ba0a2"}]: dispatch 2023-07-14T17:38:55.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T17:38:55.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/4084923967' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77c095e1-8805-435b-bb0e-1ff3ef4ba0a2"}]': finished 2023-07-14T17:38:55.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T17:38:55.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T17:38:55.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T17:38:55.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T17:38:55.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:38:55.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:38:55.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:54 smithi178 ceph-mon[129368]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T17:38:55.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-14T17:38:55.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T17:38:55.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T17:38:55.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:38:55.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:38:55.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T17:38:55.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T17:38:55.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:38:55.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:38:55.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/4084923967' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77c095e1-8805-435b-bb0e-1ff3ef4ba0a2"}]: dispatch 2023-07-14T17:38:55.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T17:38:55.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/4084923967' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77c095e1-8805-435b-bb0e-1ff3ef4ba0a2"}]': finished 2023-07-14T17:38:55.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T17:38:55.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T17:38:55.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T17:38:55.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T17:38:55.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:38:55.197 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:38:55.198 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:54 smithi088 ceph-mon[127465]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T17:38:55.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 sudo[136261]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T17:38:55.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 sudo[136261]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T17:38:55.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 sudo[136261]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T17:38:55.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 sudo[136261]: pam_unix(sudo:session): session closed for user root 2023-07-14T17:38:55.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-14T17:38:55.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T17:38:55.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T17:38:55.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:38:55.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:38:55.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T17:38:55.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T17:38:55.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:38:55.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:38:55.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/4084923967' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "77c095e1-8805-435b-bb0e-1ff3ef4ba0a2"}]: dispatch 2023-07-14T17:38:55.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T17:38:55.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/4084923967' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "77c095e1-8805-435b-bb0e-1ff3ef4ba0a2"}]': finished 2023-07-14T17:38:55.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T17:38:55.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T17:38:55.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T17:38:55.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T17:38:55.230 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T17:38:55.230 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T17:38:55.230 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:54 smithi107 ceph-mon[128999]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T17:38:56.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:55 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3738998022' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:38:56.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:55 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3738998022' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:38:56.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:55 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3738998022' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:38:57.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:56 smithi178 ceph-mon[129368]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T17:38:57.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:56 smithi088 ceph-mon[127465]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T17:38:57.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:56 smithi107 ceph-mon[128999]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T17:38:59.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:38:58 smithi178 ceph-mon[129368]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T17:38:59.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:38:58 smithi088 ceph-mon[127465]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T17:38:59.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:38:58 smithi107 ceph-mon[128999]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T17:39:01.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:00 smithi178 ceph-mon[129368]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-07-14T17:39:01.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:00 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T17:39:01.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:00 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:01.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:00 smithi088 ceph-mon[127465]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-07-14T17:39:01.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:00 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T17:39:01.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:00 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:01.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:00 smithi107 ceph-mon[128999]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-07-14T17:39:01.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:00 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T17:39:01.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:00 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:02.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:01 smithi178 ceph-mon[129368]: Deploying daemon osd.3 on smithi088 2023-07-14T17:39:02.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:01 smithi088 ceph-mon[127465]: Deploying daemon osd.3 on smithi088 2023-07-14T17:39:02.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:01 smithi107 ceph-mon[128999]: Deploying daemon osd.3 on smithi088 2023-07-14T17:39:03.161 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:02 smithi088 ceph-mon[127465]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2023-07-14T17:39:03.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:02 smithi178 ceph-mon[129368]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2023-07-14T17:39:03.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:02 smithi107 ceph-mon[128999]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2023-07-14T17:39:05.131 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:04 smithi088 ceph-mon[127465]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2023-07-14T17:39:05.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:04 smithi178 ceph-mon[129368]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2023-07-14T17:39:05.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:04 smithi107 ceph-mon[128999]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2023-07-14T17:39:06.261 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:06 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:06.261 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:06 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:06.262 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:06 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:39:06.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:06 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:06.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:06 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:06.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:06 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:39:06.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:06 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:06.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:06 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:06.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:06 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:39:07.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:07 smithi088 ceph-mon[127465]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T17:39:07.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:07 smithi107 ceph-mon[128999]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T17:39:07.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:07 smithi178 ceph-mon[129368]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T17:39:07.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:07.863+0000 7f7864ff9700 1 -- 172.21.15.88:0/4109691264 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7868060a80 con 0x7f78500a1140 2023-07-14T17:39:07.848 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 3 on host 'smithi088' 2023-07-14T17:39:07.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:07.866+0000 7f786e185700 1 -- 172.21.15.88:0/4109691264 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f78500a1140 msgr2=0x7f78500a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:39:07.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:07.866+0000 7f786e185700 1 --2- 172.21.15.88:0/4109691264 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f78500a1140 0x7f78500a35f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78680fc3e0 tx=0x7f785800a300 comp rx=0 tx=0).stop 2023-07-14T17:39:07.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:07.866+0000 7f786e185700 1 -- 172.21.15.88:0/4109691264 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78680fe380 msgr2=0x7f78680fbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:39:07.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:07.866+0000 7f786e185700 1 --2- 172.21.15.88:0/4109691264 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78680fe380 0x7f78680fbe60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f785c004730 tx=0x7f785c004760 comp rx=0 tx=0).stop 2023-07-14T17:39:07.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:07.866+0000 7f786e185700 1 -- 172.21.15.88:0/4109691264 shutdown_connections 2023-07-14T17:39:07.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:07.866+0000 7f786e185700 1 --2- 172.21.15.88:0/4109691264 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78680fff10 0x7f786819cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:07.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:07.866+0000 7f786e185700 1 --2- 172.21.15.88:0/4109691264 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f78500a1140 0x7f78500a35f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:07.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:07.866+0000 7f786e185700 1 --2- 172.21.15.88:0/4109691264 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f78680fe380 0x7f78680fbe60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:07.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:07.866+0000 7f786e185700 1 --2- 172.21.15.88:0/4109691264 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78680ff580 0x7f786819a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:07.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:07.866+0000 7f786e185700 1 -- 172.21.15.88:0/4109691264 >> 172.21.15.88:0/4109691264 conn(0x7f78680f99b0 msgr2=0x7f7868103040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:39:07.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:07.866+0000 7f786e185700 1 -- 172.21.15.88:0/4109691264 shutdown_connections 2023-07-14T17:39:07.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:07.866+0000 7f786e185700 1 -- 172.21.15.88:0/4109691264 wait complete. 2023-07-14T17:39:08.238 DEBUG:teuthology.orchestra.run.smithi088:osd.3> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.3.service 2023-07-14T17:39:08.240 INFO:tasks.cephadm:Deploying osd.4 on smithi107 with /dev/vg_nvme/lv_3... 2023-07-14T17:39:08.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T17:39:08.270 INFO:journalctl@ceph.osd.3.smithi088.stdout:-- Logs begin at Fri 2023-07-14 17:14:35 UTC. -- 2023-07-14T17:39:08.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:08 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:08.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:08 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:08.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:08 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:08.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:08 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:08.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:08 smithi178 ceph-mon[129368]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-14T17:39:08.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:08 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:08.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:08 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:08.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:08 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:08.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:08 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:08.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:08 smithi088 ceph-mon[127465]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-14T17:39:08.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:08 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:08.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:08 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:08.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:08 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:08.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:08 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:08.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:08 smithi107 ceph-mon[128999]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-14T17:39:08.809 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T17:39:09.444 INFO:journalctl@ceph.osd.3.smithi088.stdout:Jul 14 17:39:09 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-3[145776]: 2023-07-14T17:39:09.225+0000 7f72470cc640 -1 osd.3 0 log_to_monitors true 2023-07-14T17:39:09.825 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:09 smithi107 ceph-mon[128999]: from='osd.3 [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T17:39:09.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:09 smithi088 ceph-mon[127465]: from='osd.3 [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T17:39:10.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:09 smithi178 ceph-mon[129368]: from='osd.3 [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T17:39:10.301 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T17:39:10.302 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T17:39:10.302 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2023-07-14T17:39:10.302 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2023-07-14T17:39:10.303 INFO:teuthology.orchestra.run.smithi107.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110672 s, 947 MB/s 2023-07-14T17:39:10.303 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2023-07-14T17:39:10.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi107:vg_nvme/lv_3 2023-07-14T17:39:10.944 INFO:journalctl@ceph.osd.3.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-3[145776]: 2023-07-14T17:39:10.834+0000 7f723c744700 -1 osd.3 0 waiting for initial osdmap 2023-07-14T17:39:10.945 INFO:journalctl@ceph.osd.3.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-3[145776]: 2023-07-14T17:39:10.839+0000 7f7236d37700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T17:39:10.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-mon[127465]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T17:39:10.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-mon[127465]: from='osd.3 [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T17:39:10.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-mon[127465]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T17:39:10.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T17:39:10.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-mon[127465]: from='osd.3 [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T17:39:10.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:10.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:10.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:39:10.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:39:10.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:10.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:10.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:39:10.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:10.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:39:10.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:10.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:39:10.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:10 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:10.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:10 smithi107 ceph-mon[128999]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T17:39:10.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:10 smithi107 ceph-mon[128999]: from='osd.3 [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T17:39:10.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:10 smithi107 ceph-mon[128999]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T17:39:10.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:10 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T17:39:10.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:10 smithi107 ceph-mon[128999]: from='osd.3 [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T17:39:10.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:10 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:10.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:10 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:10.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:10 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:39:10.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:10 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:39:10.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:10 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:10.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:10 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:10.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:10 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:39:10.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:10 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:10.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:10 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:39:10.979 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:10 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:10.979 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:10 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:39:10.979 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:10 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:11.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:10 smithi178 ceph-mon[129368]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T17:39:11.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:10 smithi178 ceph-mon[129368]: from='osd.3 [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T17:39:11.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:10 smithi178 ceph-mon[129368]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T17:39:11.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:10 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T17:39:11.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:10 smithi178 ceph-mon[129368]: from='osd.3 [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T17:39:11.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:10 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:11.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:10 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:11.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:10 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:39:11.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:10 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:39:11.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:10 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:11.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:10 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:11.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:10 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:39:11.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:10 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:11.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:10 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:39:11.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:10 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:11.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:10 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:39:11.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:10 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:11.316 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T17:39:12.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:11 smithi178 ceph-mon[129368]: Detected new or changed devices on smithi088 2023-07-14T17:39:12.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:11 smithi178 ceph-mon[129368]: Adjusting osd_memory_target on smithi088 to 8584M 2023-07-14T17:39:12.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:11 smithi178 ceph-mon[129368]: from='osd.3 [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T17:39:12.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:11 smithi178 ceph-mon[129368]: osdmap e24: 4 total, 3 up, 4 in 2023-07-14T17:39:12.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:11 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T17:39:12.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:11 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T17:39:12.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:11 smithi088 ceph-mon[127465]: Detected new or changed devices on smithi088 2023-07-14T17:39:12.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:11 smithi088 ceph-mon[127465]: Adjusting osd_memory_target on smithi088 to 8584M 2023-07-14T17:39:12.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:11 smithi088 ceph-mon[127465]: from='osd.3 [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T17:39:12.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:11 smithi088 ceph-mon[127465]: osdmap e24: 4 total, 3 up, 4 in 2023-07-14T17:39:12.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:11 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T17:39:12.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:11 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T17:39:12.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:11 smithi107 ceph-mon[128999]: Detected new or changed devices on smithi088 2023-07-14T17:39:12.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:11 smithi107 ceph-mon[128999]: Adjusting osd_memory_target on smithi088 to 8584M 2023-07-14T17:39:12.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:11 smithi107 ceph-mon[128999]: from='osd.3 [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T17:39:12.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:11 smithi107 ceph-mon[128999]: osdmap e24: 4 total, 3 up, 4 in 2023-07-14T17:39:12.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:11 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T17:39:12.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:11 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T17:39:12.499 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.514+0000 7f76f785c700 1 -- 172.21.15.107:0/1438326499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f00ffcc0 msgr2=0x7f76f01000d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:39:12.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.514+0000 7f76f785c700 1 --2- 172.21.15.107:0/1438326499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f00ffcc0 0x7f76f01000d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009990 tx=0x7f76ec009ca0 comp rx=0 tx=0).stop 2023-07-14T17:39:12.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.514+0000 7f76f785c700 1 -- 172.21.15.107:0/1438326499 shutdown_connections 2023-07-14T17:39:12.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.514+0000 7f76f785c700 1 --2- 172.21.15.107:0/1438326499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76f0101850 0x7f76f0108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:12.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.514+0000 7f76f785c700 1 --2- 172.21.15.107:0/1438326499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f00ffcc0 0x7f76f01000d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:12.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.514+0000 7f76f785c700 1 --2- 172.21.15.107:0/1438326499 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76f0100ec0 0x7f76f0101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:12.500 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.514+0000 7f76f785c700 1 -- 172.21.15.107:0/1438326499 >> 172.21.15.107:0/1438326499 conn(0x7f76f00fb2d0 msgr2=0x7f76f00fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:39:12.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.514+0000 7f76f785c700 1 -- 172.21.15.107:0/1438326499 shutdown_connections 2023-07-14T17:39:12.501 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.514+0000 7f76f785c700 1 -- 172.21.15.107:0/1438326499 wait complete. 2023-07-14T17:39:12.502 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.515+0000 7f76f785c700 1 Processor -- start 2023-07-14T17:39:12.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.515+0000 7f76f785c700 1 -- start start 2023-07-14T17:39:12.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.515+0000 7f76f785c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f00ffcc0 0x7f76f01958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:39:12.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.515+0000 7f76f785c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76f0100ec0 0x7f76f0195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:39:12.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.515+0000 7f76f785c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76f0101850 0x7f76f019ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:39:12.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.515+0000 7f76f785c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f019d480 con 0x7f76f0100ec0 2023-07-14T17:39:12.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.515+0000 7f76f785c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f019d5c0 con 0x7f76f00ffcc0 2023-07-14T17:39:12.503 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.515+0000 7f76f785c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f019d750 con 0x7f76f0101850 2023-07-14T17:39:12.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.515+0000 7f76f55f8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f00ffcc0 0x7f76f01958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:39:12.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.515+0000 7f76f55f8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f00ffcc0 0x7f76f01958f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42516/0 (socket says 172.21.15.107:42516) 2023-07-14T17:39:12.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.515+0000 7f76f55f8700 1 -- 172.21.15.107:0/257858310 learned_addr learned my addr 172.21.15.107:0/257858310 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-07-14T17:39:12.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.516+0000 7f76f4df7700 1 --2- 172.21.15.107:0/257858310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76f0100ec0 0x7f76f0195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:39:12.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.516+0000 7f76f5df9700 1 --2- 172.21.15.107:0/257858310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76f0101850 0x7f76f019ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:39:12.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.516+0000 7f76f55f8700 1 -- 172.21.15.107:0/257858310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76f0101850 msgr2=0x7f76f019ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:39:12.504 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.516+0000 7f76f55f8700 1 --2- 172.21.15.107:0/257858310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76f0101850 0x7f76f019ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:12.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.516+0000 7f76f55f8700 1 -- 172.21.15.107:0/257858310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76f0100ec0 msgr2=0x7f76f0195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:39:12.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.516+0000 7f76f55f8700 1 --2- 172.21.15.107:0/257858310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76f0100ec0 0x7f76f0195e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:12.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.516+0000 7f76f55f8700 1 -- 172.21.15.107:0/257858310 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec0095f0 con 0x7f76f00ffcc0 2023-07-14T17:39:12.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.516+0000 7f76f5df9700 1 --2- 172.21.15.107:0/257858310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76f0101850 0x7f76f019ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:39:12.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.516+0000 7f76f55f8700 1 --2- 172.21.15.107:0/257858310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f00ffcc0 0x7f76f01958f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f76ec00c010 tx=0x7f76ec005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:39:12.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.516+0000 7f76e67fc700 1 -- 172.21.15.107:0/257858310 <== mon.2 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ec01d070 con 0x7f76f00ffcc0 2023-07-14T17:39:12.505 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.516+0000 7f76f785c700 1 -- 172.21.15.107:0/257858310 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f019d9d0 con 0x7f76f00ffcc0 2023-07-14T17:39:12.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.516+0000 7f76e67fc700 1 -- 172.21.15.107:0/257858310 <== mon.2 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f76ec004100 con 0x7f76f00ffcc0 2023-07-14T17:39:12.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.516+0000 7f76e67fc700 1 -- 172.21.15.107:0/257858310 <== mon.2 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ec005290 con 0x7f76f00ffcc0 2023-07-14T17:39:12.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.516+0000 7f76f785c700 1 -- 172.21.15.107:0/257858310 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f019de40 con 0x7f76f00ffcc0 2023-07-14T17:39:12.506 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.517+0000 7f76f785c700 1 -- 172.21.15.107:0/257858310 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f0107360 con 0x7f76f00ffcc0 2023-07-14T17:39:12.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.518+0000 7f76e67fc700 1 -- 172.21.15.107:0/257858310 <== mon.2 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f76ec0047e0 con 0x7f76f00ffcc0 2023-07-14T17:39:12.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.518+0000 7f76e67fc700 1 --2- 172.21.15.107:0/257858310 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f76dc0a1150 0x7f76dc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:39:12.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.519+0000 7f76e67fc700 1 -- 172.21.15.107:0/257858310 <== mon.2 v2:172.21.15.107:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 3702+0+0 (secure 0 0 0) 0x7f76ec0cc7c0 con 0x7f76f00ffcc0 2023-07-14T17:39:12.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.519+0000 7f76f4df7700 1 --2- 172.21.15.107:0/257858310 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f76dc0a1150 0x7f76dc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:39:12.507 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.519+0000 7f76f4df7700 1 --2- 172.21.15.107:0/257858310 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f76dc0a1150 0x7f76dc0a3600 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f76e0005fd0 tx=0x7f76e000a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:39:12.508 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.523+0000 7f76e67fc700 1 -- 172.21.15.107:0/257858310 <== mon.2 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f76ec0971a0 con 0x7f76f00ffcc0 2023-07-14T17:39:12.700 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:12.714+0000 7f76f785c700 1 -- 172.21.15.107:0/257858310 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f76f0060a80 con 0x7f76dc0a1150 2023-07-14T17:39:13.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:12 smithi178 ceph-mon[129368]: purged_snaps scrub starts 2023-07-14T17:39:13.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:12 smithi178 ceph-mon[129368]: purged_snaps scrub ok 2023-07-14T17:39:13.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:12 smithi178 ceph-mon[129368]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T17:39:13.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:12 smithi178 ceph-mon[129368]: osd.3 [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618] boot 2023-07-14T17:39:13.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:12 smithi178 ceph-mon[129368]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T17:39:13.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:12 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T17:39:13.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:12 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:39:13.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:12 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:39:13.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:12 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:13.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:12 smithi178 ceph-mon[129368]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T17:39:13.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:12 smithi088 ceph-mon[127465]: purged_snaps scrub starts 2023-07-14T17:39:13.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:12 smithi088 ceph-mon[127465]: purged_snaps scrub ok 2023-07-14T17:39:13.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:12 smithi088 ceph-mon[127465]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T17:39:13.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:12 smithi088 ceph-mon[127465]: osd.3 [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618] boot 2023-07-14T17:39:13.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:12 smithi088 ceph-mon[127465]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T17:39:13.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:12 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T17:39:13.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:12 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:39:13.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:12 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:39:13.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:12 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:13.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:12 smithi088 ceph-mon[127465]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T17:39:13.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:12 smithi107 ceph-mon[128999]: purged_snaps scrub starts 2023-07-14T17:39:13.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:12 smithi107 ceph-mon[128999]: purged_snaps scrub ok 2023-07-14T17:39:13.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:12 smithi107 ceph-mon[128999]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T17:39:13.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:12 smithi107 ceph-mon[128999]: osd.3 [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618] boot 2023-07-14T17:39:13.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:12 smithi107 ceph-mon[128999]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T17:39:13.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:12 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T17:39:13.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:12 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:39:13.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:12 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:39:13.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:12 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:13.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:12 smithi107 ceph-mon[128999]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T17:39:14.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:13 smithi178 ceph-mon[129368]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:39:14.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:13 smithi088 ceph-mon[127465]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:39:14.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:13 smithi107 ceph-mon[128999]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:39:15.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:14 smithi178 ceph-mon[129368]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T17:39:15.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:14 smithi178 ceph-mon[129368]: osdmap e27: 4 total, 4 up, 4 in 2023-07-14T17:39:15.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:14 smithi088 ceph-mon[127465]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T17:39:15.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:14 smithi088 ceph-mon[127465]: osdmap e27: 4 total, 4 up, 4 in 2023-07-14T17:39:15.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:14 smithi107 ceph-mon[128999]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T17:39:15.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:14 smithi107 ceph-mon[128999]: osdmap e27: 4 total, 4 up, 4 in 2023-07-14T17:39:16.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:15 smithi178 ceph-mon[129368]: osdmap e28: 4 total, 4 up, 4 in 2023-07-14T17:39:16.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:15 smithi178 ceph-mon[129368]: from='client.? 172.21.15.107:0/1990170874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae0f806e-a7ef-445d-a7d4-14823157b841"}]: dispatch 2023-07-14T17:39:16.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:15 smithi178 ceph-mon[129368]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae0f806e-a7ef-445d-a7d4-14823157b841"}]: dispatch 2023-07-14T17:39:16.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:15 smithi178 ceph-mon[129368]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae0f806e-a7ef-445d-a7d4-14823157b841"}]': finished 2023-07-14T17:39:16.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:15 smithi178 ceph-mon[129368]: osdmap e29: 5 total, 4 up, 5 in 2023-07-14T17:39:16.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:15 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T17:39:16.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:15 smithi178 ceph-mon[129368]: from='client.? 172.21.15.107:0/3415336517' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:39:16.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:15 smithi088 ceph-mon[127465]: osdmap e28: 4 total, 4 up, 4 in 2023-07-14T17:39:16.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:15 smithi088 ceph-mon[127465]: from='client.? 172.21.15.107:0/1990170874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae0f806e-a7ef-445d-a7d4-14823157b841"}]: dispatch 2023-07-14T17:39:16.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:15 smithi088 ceph-mon[127465]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae0f806e-a7ef-445d-a7d4-14823157b841"}]: dispatch 2023-07-14T17:39:16.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:15 smithi088 ceph-mon[127465]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae0f806e-a7ef-445d-a7d4-14823157b841"}]': finished 2023-07-14T17:39:16.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:15 smithi088 ceph-mon[127465]: osdmap e29: 5 total, 4 up, 5 in 2023-07-14T17:39:16.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:15 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T17:39:16.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:15 smithi088 ceph-mon[127465]: from='client.? 172.21.15.107:0/3415336517' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:39:16.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:15 smithi107 ceph-mon[128999]: osdmap e28: 4 total, 4 up, 4 in 2023-07-14T17:39:16.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:15 smithi107 ceph-mon[128999]: from='client.? 172.21.15.107:0/1990170874' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae0f806e-a7ef-445d-a7d4-14823157b841"}]: dispatch 2023-07-14T17:39:16.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:15 smithi107 ceph-mon[128999]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ae0f806e-a7ef-445d-a7d4-14823157b841"}]: dispatch 2023-07-14T17:39:16.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:15 smithi107 ceph-mon[128999]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ae0f806e-a7ef-445d-a7d4-14823157b841"}]': finished 2023-07-14T17:39:16.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:15 smithi107 ceph-mon[128999]: osdmap e29: 5 total, 4 up, 5 in 2023-07-14T17:39:16.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:15 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T17:39:16.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:15 smithi107 ceph-mon[128999]: from='client.? 172.21.15.107:0/3415336517' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:39:17.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:16 smithi178 ceph-mon[129368]: pgmap v95: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T17:39:17.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:16 smithi088 ceph-mon[127465]: pgmap v95: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T17:39:17.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:16 smithi107 ceph-mon[128999]: pgmap v95: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T17:39:19.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:18 smithi178 ceph-mon[129368]: pgmap v96: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2023-07-14T17:39:19.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:18 smithi088 ceph-mon[127465]: pgmap v96: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2023-07-14T17:39:19.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:18 smithi107 ceph-mon[128999]: pgmap v96: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 115 KiB/s, 0 objects/s recovering 2023-07-14T17:39:20.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:20 smithi107 ceph-mon[128999]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T17:39:21.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:20 smithi178 ceph-mon[129368]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T17:39:21.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:20 smithi088 ceph-mon[127465]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T17:39:23.140 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:22 smithi107 ceph-mon[128999]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T17:39:23.140 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:22 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T17:39:23.140 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:22 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:23.141 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:22 smithi107 ceph-mon[128999]: Deploying daemon osd.4 on smithi107 2023-07-14T17:39:23.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:22 smithi178 ceph-mon[129368]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T17:39:23.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:22 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T17:39:23.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:22 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:23.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:22 smithi178 ceph-mon[129368]: Deploying daemon osd.4 on smithi107 2023-07-14T17:39:23.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:22 smithi088 ceph-mon[127465]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T17:39:23.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:22 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T17:39:23.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:22 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:23.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:22 smithi088 ceph-mon[127465]: Deploying daemon osd.4 on smithi107 2023-07-14T17:39:25.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:24 smithi178 ceph-mon[129368]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T17:39:25.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:24 smithi088 ceph-mon[127465]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T17:39:25.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:24 smithi107 ceph-mon[128999]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T17:39:26.895 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:26 smithi107 ceph-mon[128999]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T17:39:26.896 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:26 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:26.896 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:26 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:26.896 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:26 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:39:26.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:26 smithi178 ceph-mon[129368]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T17:39:26.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:26 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:26.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:26 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:26.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:26 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:39:26.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:26 smithi088 ceph-mon[127465]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T17:39:26.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:26 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:26.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:26 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:26.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:26 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:39:29.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:28 smithi178 ceph-mon[129368]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T17:39:29.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:28 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:29.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:28 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:29.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:28 smithi088 ceph-mon[127465]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T17:39:29.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:28 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:29.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:28 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:29.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:28 smithi107 ceph-mon[128999]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T17:39:29.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:29.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:28 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:29.310 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:29.324+0000 7f76e67fc700 1 -- 172.21.15.107:0/257858310 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f76f0060a80 con 0x7f76dc0a1150 2023-07-14T17:39:29.310 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 4 on host 'smithi107' 2023-07-14T17:39:29.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:29.327+0000 7f76f785c700 1 -- 172.21.15.107:0/257858310 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f76dc0a1150 msgr2=0x7f76dc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:39:29.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:29.327+0000 7f76f785c700 1 --2- 172.21.15.107:0/257858310 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f76dc0a1150 0x7f76dc0a3600 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f76e0005fd0 tx=0x7f76e000a560 comp rx=0 tx=0).stop 2023-07-14T17:39:29.312 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:29.327+0000 7f76f785c700 1 -- 172.21.15.107:0/257858310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f00ffcc0 msgr2=0x7f76f01958f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:39:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:29.327+0000 7f76f785c700 1 --2- 172.21.15.107:0/257858310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f00ffcc0 0x7f76f01958f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f76ec00c010 tx=0x7f76ec005ef0 comp rx=0 tx=0).stop 2023-07-14T17:39:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:29.327+0000 7f76f785c700 1 -- 172.21.15.107:0/257858310 shutdown_connections 2023-07-14T17:39:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:29.327+0000 7f76f785c700 1 --2- 172.21.15.107:0/257858310 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f76f0101850 0x7f76f019ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:29.327+0000 7f76f785c700 1 --2- 172.21.15.107:0/257858310 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f76f00ffcc0 0x7f76f01958f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:29.327+0000 7f76f785c700 1 --2- 172.21.15.107:0/257858310 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f76dc0a1150 0x7f76dc0a3600 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:29.327+0000 7f76f785c700 1 --2- 172.21.15.107:0/257858310 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f76f0100ec0 0x7f76f0195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:29.327+0000 7f76f785c700 1 -- 172.21.15.107:0/257858310 >> 172.21.15.107:0/257858310 conn(0x7f76f00fb2d0 msgr2=0x7f76f01044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:39:29.313 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:29.327+0000 7f76f785c700 1 -- 172.21.15.107:0/257858310 shutdown_connections 2023-07-14T17:39:29.314 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:39:29.327+0000 7f76f785c700 1 -- 172.21.15.107:0/257858310 wait complete. 2023-07-14T17:39:29.711 DEBUG:teuthology.orchestra.run.smithi107:osd.4> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.4.service 2023-07-14T17:39:29.713 INFO:tasks.cephadm:Deploying osd.5 on smithi178 with /dev/vg_nvme/lv_3... 2023-07-14T17:39:29.713 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T17:39:29.748 INFO:journalctl@ceph.osd.4.smithi107.stdout:-- Logs begin at Fri 2023-07-14 17:14:00 UTC. -- 2023-07-14T17:39:30.275 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T17:39:30.525 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:30 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:30.525 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:30 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:30.564 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:30 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:30.564 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:30 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:30.564 INFO:journalctl@ceph.osd.4.smithi107.stdout:Jul 14 17:39:30 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-4[139372]: 2023-07-14T17:39:30.460+0000 7fbddad2f640 -1 osd.4 0 log_to_monitors true 2023-07-14T17:39:30.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:30 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:30.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:30 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:31.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:31 smithi107 ceph-mon[128999]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T17:39:31.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:31 smithi107 ceph-mon[128999]: from='osd.4 [v2:172.21.15.107:6808/2380989846,v1:172.21.15.107:6809/2380989846]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T17:39:31.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:31 smithi107 ceph-mon[128999]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T17:39:31.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:31 smithi178 ceph-mon[129368]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T17:39:31.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:31 smithi178 ceph-mon[129368]: from='osd.4 [v2:172.21.15.107:6808/2380989846,v1:172.21.15.107:6809/2380989846]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T17:39:31.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:31 smithi178 ceph-mon[129368]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T17:39:31.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:31 smithi088 ceph-mon[127465]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T17:39:31.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:31 smithi088 ceph-mon[127465]: from='osd.4 [v2:172.21.15.107:6808/2380989846,v1:172.21.15.107:6809/2380989846]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T17:39:31.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:31 smithi088 ceph-mon[127465]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T17:39:31.844 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T17:39:31.845 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T17:39:31.845 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2023-07-14T17:39:31.845 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2023-07-14T17:39:31.845 INFO:teuthology.orchestra.run.smithi178.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00813376 s, 1.3 GB/s 2023-07-14T17:39:31.845 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 2023-07-14T17:39:31.845 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2023-07-14T17:39:32.226 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi178:vg_nvme/lv_3 2023-07-14T17:39:32.477 INFO:journalctl@ceph.osd.4.smithi107.stdout:Jul 14 17:39:32 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-4[139372]: 2023-07-14T17:39:32.302+0000 7fbdd03a7700 -1 osd.4 0 waiting for initial osdmap 2023-07-14T17:39:32.477 INFO:journalctl@ceph.osd.4.smithi107.stdout:Jul 14 17:39:32 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-4[139372]: 2023-07-14T17:39:32.309+0000 7fbdc9197700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T17:39:32.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:32 smithi107 ceph-mon[128999]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T17:39:32.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:32 smithi107 ceph-mon[128999]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T17:39:32.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T17:39:32.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:32 smithi107 ceph-mon[128999]: from='osd.4 [v2:172.21.15.107:6808/2380989846,v1:172.21.15.107:6809/2380989846]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:39:32.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:32 smithi107 ceph-mon[128999]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:39:32.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:32 smithi107 ceph-mon[128999]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T17:39:32.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:32 smithi107 ceph-mon[128999]: Detected new or changed devices on smithi107 2023-07-14T17:39:32.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:32.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:32.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:39:32.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:39:32.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:32 smithi107 ceph-mon[128999]: Adjusting osd_memory_target on smithi107 to 8584M 2023-07-14T17:39:32.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:32.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:32.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:39:32.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:32.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:32 smithi178 ceph-mon[129368]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T17:39:32.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:32 smithi178 ceph-mon[129368]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T17:39:32.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T17:39:32.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:32 smithi178 ceph-mon[129368]: from='osd.4 [v2:172.21.15.107:6808/2380989846,v1:172.21.15.107:6809/2380989846]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:39:32.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:32 smithi178 ceph-mon[129368]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:39:32.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:32 smithi178 ceph-mon[129368]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T17:39:32.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:32 smithi178 ceph-mon[129368]: Detected new or changed devices on smithi107 2023-07-14T17:39:32.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:32.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:32.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:39:32.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:39:32.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:32 smithi178 ceph-mon[129368]: Adjusting osd_memory_target on smithi107 to 8584M 2023-07-14T17:39:32.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:32.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:32.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:39:32.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:32.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:32 smithi088 ceph-mon[127465]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T17:39:32.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:32 smithi088 ceph-mon[127465]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T17:39:32.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T17:39:32.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:32 smithi088 ceph-mon[127465]: from='osd.4 [v2:172.21.15.107:6808/2380989846,v1:172.21.15.107:6809/2380989846]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:39:32.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:32 smithi088 ceph-mon[127465]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:39:32.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:32 smithi088 ceph-mon[127465]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T17:39:32.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:32 smithi088 ceph-mon[127465]: Detected new or changed devices on smithi107 2023-07-14T17:39:32.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:32.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:32.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:39:32.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:39:32.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:32 smithi088 ceph-mon[127465]: Adjusting osd_memory_target on smithi107 to 8584M 2023-07-14T17:39:32.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:32.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:32.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:39:32.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:32.781 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T17:39:33.546 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:33 smithi178 ceph-mon[129368]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2023-07-14T17:39:33.547 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:33 smithi178 ceph-mon[129368]: osdmap e31: 5 total, 4 up, 5 in 2023-07-14T17:39:33.547 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:33 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T17:39:33.547 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:33 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T17:39:33.547 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:33 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:39:33.547 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:33 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:33.547 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:33 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:39:33.547 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:33 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:33.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:33 smithi088 ceph-mon[127465]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2023-07-14T17:39:33.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:33 smithi088 ceph-mon[127465]: osdmap e31: 5 total, 4 up, 5 in 2023-07-14T17:39:33.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:33 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T17:39:33.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:33 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T17:39:33.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:33 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:39:33.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:33 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:33.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:33 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:39:33.696 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:33 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:33.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:33 smithi107 ceph-mon[128999]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2023-07-14T17:39:33.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:33 smithi107 ceph-mon[128999]: osdmap e31: 5 total, 4 up, 5 in 2023-07-14T17:39:33.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:33 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T17:39:33.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:33 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T17:39:33.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:33 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:39:33.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:33 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:33.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:33 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:39:33.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:33 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:33.974 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.987+0000 7f0b09697700 1 -- 172.21.15.178:0/4253786244 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b040ffdb0 msgr2=0x7f0b04100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:39:33.975 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.987+0000 7f0b09697700 1 --2- 172.21.15.178:0/4253786244 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b040ffdb0 0x7f0b04100200 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0af4009990 tx=0x7f0af4009ca0 comp rx=0 tx=0).stop 2023-07-14T17:39:33.975 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.987+0000 7f0b09697700 1 -- 172.21.15.178:0/4253786244 shutdown_connections 2023-07-14T17:39:33.975 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.987+0000 7f0b09697700 1 --2- 172.21.15.178:0/4253786244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b04100740 0x7f0b04107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:33.975 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.987+0000 7f0b09697700 1 --2- 172.21.15.178:0/4253786244 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b040febb0 0x7f0b040fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:33.975 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.987+0000 7f0b09697700 1 --2- 172.21.15.178:0/4253786244 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b040ffdb0 0x7f0b04100200 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:33.976 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.987+0000 7f0b09697700 1 -- 172.21.15.178:0/4253786244 >> 172.21.15.178:0/4253786244 conn(0x7f0b040fa1e0 msgr2=0x7f0b040fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:39:33.976 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.987+0000 7f0b09697700 1 -- 172.21.15.178:0/4253786244 shutdown_connections 2023-07-14T17:39:33.976 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.987+0000 7f0b09697700 1 -- 172.21.15.178:0/4253786244 wait complete. 2023-07-14T17:39:33.976 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.988+0000 7f0b09697700 1 Processor -- start 2023-07-14T17:39:33.976 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.988+0000 7f0b09697700 1 -- start start 2023-07-14T17:39:33.976 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.988+0000 7f0b09697700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b040febb0 0x7f0b04068c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:39:33.976 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.988+0000 7f0b09697700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b040ffdb0 0x7f0b0419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:39:33.977 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.988+0000 7f0b09697700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b04100740 0x7f0b0419cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:39:33.977 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.988+0000 7f0b09697700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0419d420 con 0x7f0b040febb0 2023-07-14T17:39:33.977 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.988+0000 7f0b09697700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0419d590 con 0x7f0b04100740 2023-07-14T17:39:33.977 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.988+0000 7f0b09697700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0419d750 con 0x7f0b040ffdb0 2023-07-14T17:39:33.977 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.988+0000 7f0b027fc700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b040ffdb0 0x7f0b0419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:39:33.977 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.988+0000 7f0b027fc700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b040ffdb0 0x7f0b0419a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:54732/0 (socket says 172.21.15.178:54732) 2023-07-14T17:39:33.977 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.988+0000 7f0b027fc700 1 -- 172.21.15.178:0/3742860083 learned_addr learned my addr 172.21.15.178:0/3742860083 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-07-14T17:39:33.978 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.988+0000 7f0b037fe700 1 --2- 172.21.15.178:0/3742860083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b04100740 0x7f0b0419cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:39:33.978 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.988+0000 7f0b02ffd700 1 --2- 172.21.15.178:0/3742860083 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b040febb0 0x7f0b04068c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:39:33.978 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.989+0000 7f0b027fc700 1 -- 172.21.15.178:0/3742860083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b04100740 msgr2=0x7f0b0419cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:39:33.978 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.989+0000 7f0b027fc700 1 --2- 172.21.15.178:0/3742860083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b04100740 0x7f0b0419cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:33.978 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.989+0000 7f0b027fc700 1 -- 172.21.15.178:0/3742860083 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b040febb0 msgr2=0x7f0b04068c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:39:33.978 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.989+0000 7f0b027fc700 1 --2- 172.21.15.178:0/3742860083 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b040febb0 0x7f0b04068c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:33.978 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.989+0000 7f0b027fc700 1 -- 172.21.15.178:0/3742860083 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af40095f0 con 0x7f0b040ffdb0 2023-07-14T17:39:33.978 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.989+0000 7f0b027fc700 1 --2- 172.21.15.178:0/3742860083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b040ffdb0 0x7f0b0419a640 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0af4006010 tx=0x7f0af4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:39:33.979 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.989+0000 7f0afbfff700 1 -- 172.21.15.178:0/3742860083 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af401d070 con 0x7f0b040ffdb0 2023-07-14T17:39:33.979 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.989+0000 7f0afbfff700 1 -- 172.21.15.178:0/3742860083 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0af4004080 con 0x7f0b040ffdb0 2023-07-14T17:39:33.979 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.989+0000 7f0b09697700 1 -- 172.21.15.178:0/3742860083 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b04194550 con 0x7f0b040ffdb0 2023-07-14T17:39:33.979 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.989+0000 7f0afbfff700 1 -- 172.21.15.178:0/3742860083 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af4022c90 con 0x7f0b040ffdb0 2023-07-14T17:39:33.979 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.989+0000 7f0b09697700 1 -- 172.21.15.178:0/3742860083 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b041949f0 con 0x7f0b040ffdb0 2023-07-14T17:39:33.980 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.990+0000 7f0b09697700 1 -- 172.21.15.178:0/3742860083 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b04106240 con 0x7f0b040ffdb0 2023-07-14T17:39:33.980 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.991+0000 7f0afbfff700 1 -- 172.21.15.178:0/3742860083 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0af4022470 con 0x7f0b040ffdb0 2023-07-14T17:39:33.980 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.991+0000 7f0afbfff700 1 --2- 172.21.15.178:0/3742860083 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0af00a1150 0x7f0af00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:39:33.981 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.992+0000 7f0afbfff700 1 -- 172.21.15.178:0/3742860083 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f0af40ccb10 con 0x7f0b040ffdb0 2023-07-14T17:39:33.981 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.992+0000 7f0b02ffd700 1 --2- 172.21.15.178:0/3742860083 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0af00a1150 0x7f0af00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:39:33.981 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.992+0000 7f0b02ffd700 1 --2- 172.21.15.178:0/3742860083 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0af00a1150 0x7f0af00a3600 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0aec005fd0 tx=0x7f0aec005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:39:33.981 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:33.996+0000 7f0afbfff700 1 -- 172.21.15.178:0/3742860083 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0af405c780 con 0x7f0b040ffdb0 2023-07-14T17:39:34.171 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:34.187+0000 7f0b09697700 1 -- 172.21.15.178:0/3742860083 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b04060a80 con 0x7f0af00a1150 2023-07-14T17:39:34.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:34 smithi178 ceph-mon[129368]: purged_snaps scrub starts 2023-07-14T17:39:34.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:34 smithi178 ceph-mon[129368]: purged_snaps scrub ok 2023-07-14T17:39:34.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:34 smithi178 ceph-mon[129368]: osd.4 [v2:172.21.15.107:6808/2380989846,v1:172.21.15.107:6809/2380989846] boot 2023-07-14T17:39:34.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:34 smithi178 ceph-mon[129368]: osdmap e32: 5 total, 5 up, 5 in 2023-07-14T17:39:34.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T17:39:34.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:34 smithi178 ceph-mon[129368]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:34.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:34 smithi178 ceph-mon[129368]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:39:34.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:39:34.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:39:34.668 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:34.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:34 smithi088 ceph-mon[127465]: purged_snaps scrub starts 2023-07-14T17:39:34.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:34 smithi088 ceph-mon[127465]: purged_snaps scrub ok 2023-07-14T17:39:34.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:34 smithi088 ceph-mon[127465]: osd.4 [v2:172.21.15.107:6808/2380989846,v1:172.21.15.107:6809/2380989846] boot 2023-07-14T17:39:34.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:34 smithi088 ceph-mon[127465]: osdmap e32: 5 total, 5 up, 5 in 2023-07-14T17:39:34.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T17:39:34.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:34 smithi088 ceph-mon[127465]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:34.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:34 smithi088 ceph-mon[127465]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:39:34.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:39:34.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:39:34.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:34.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:34 smithi107 ceph-mon[128999]: purged_snaps scrub starts 2023-07-14T17:39:34.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:34 smithi107 ceph-mon[128999]: purged_snaps scrub ok 2023-07-14T17:39:34.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:34 smithi107 ceph-mon[128999]: osd.4 [v2:172.21.15.107:6808/2380989846,v1:172.21.15.107:6809/2380989846] boot 2023-07-14T17:39:34.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:34 smithi107 ceph-mon[128999]: osdmap e32: 5 total, 5 up, 5 in 2023-07-14T17:39:34.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:34 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T17:39:34.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:34 smithi107 ceph-mon[128999]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:34.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:34 smithi107 ceph-mon[128999]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:39:34.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:34 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:39:34.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:34 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:39:34.727 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:34 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:35.544 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:35 smithi178 ceph-mon[129368]: osdmap e33: 5 total, 5 up, 5 in 2023-07-14T17:39:35.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:35 smithi088 ceph-mon[127465]: osdmap e33: 5 total, 5 up, 5 in 2023-07-14T17:39:35.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:35 smithi107 ceph-mon[128999]: osdmap e33: 5 total, 5 up, 5 in 2023-07-14T17:39:36.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:36 smithi178 ceph-mon[129368]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:36.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:36 smithi178 ceph-mon[129368]: from='client.? 172.21.15.178:0/852718133' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b926d4fa-a9de-4bb6-8e00-30a2b8a35b36"}]: dispatch 2023-07-14T17:39:36.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:36 smithi178 ceph-mon[129368]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b926d4fa-a9de-4bb6-8e00-30a2b8a35b36"}]: dispatch 2023-07-14T17:39:36.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:36 smithi178 ceph-mon[129368]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b926d4fa-a9de-4bb6-8e00-30a2b8a35b36"}]': finished 2023-07-14T17:39:36.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:36 smithi178 ceph-mon[129368]: osdmap e34: 6 total, 5 up, 6 in 2023-07-14T17:39:36.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:36 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T17:39:36.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:36 smithi088 ceph-mon[127465]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:36.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:36 smithi088 ceph-mon[127465]: from='client.? 172.21.15.178:0/852718133' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b926d4fa-a9de-4bb6-8e00-30a2b8a35b36"}]: dispatch 2023-07-14T17:39:36.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:36 smithi088 ceph-mon[127465]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b926d4fa-a9de-4bb6-8e00-30a2b8a35b36"}]: dispatch 2023-07-14T17:39:36.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:36 smithi088 ceph-mon[127465]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b926d4fa-a9de-4bb6-8e00-30a2b8a35b36"}]': finished 2023-07-14T17:39:36.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:36 smithi088 ceph-mon[127465]: osdmap e34: 6 total, 5 up, 6 in 2023-07-14T17:39:36.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:36 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T17:39:36.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:36 smithi107 ceph-mon[128999]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:36.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:36 smithi107 ceph-mon[128999]: from='client.? 172.21.15.178:0/852718133' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b926d4fa-a9de-4bb6-8e00-30a2b8a35b36"}]: dispatch 2023-07-14T17:39:36.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:36 smithi107 ceph-mon[128999]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b926d4fa-a9de-4bb6-8e00-30a2b8a35b36"}]: dispatch 2023-07-14T17:39:36.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:36 smithi107 ceph-mon[128999]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b926d4fa-a9de-4bb6-8e00-30a2b8a35b36"}]': finished 2023-07-14T17:39:36.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:36 smithi107 ceph-mon[128999]: osdmap e34: 6 total, 5 up, 6 in 2023-07-14T17:39:36.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:36 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T17:39:37.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:37 smithi178 ceph-mon[129368]: from='client.? 172.21.15.178:0/3807966065' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:39:37.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:37 smithi088 ceph-mon[127465]: from='client.? 172.21.15.178:0/3807966065' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:39:37.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:37 smithi107 ceph-mon[128999]: from='client.? 172.21.15.178:0/3807966065' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:39:38.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:38 smithi178 ceph-mon[129368]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:38.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:38 smithi088 ceph-mon[127465]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:38.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:38 smithi107 ceph-mon[128999]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:41.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:40 smithi178 ceph-mon[129368]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:41.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:40 smithi088 ceph-mon[127465]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:41.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:40 smithi107 ceph-mon[128999]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:43.092 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:42 smithi178 ceph-mon[129368]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:43.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:42 smithi088 ceph-mon[127465]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:43.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:42 smithi107 ceph-mon[128999]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:44.091 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T17:39:44.091 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:44.091 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:43 smithi178 ceph-mon[129368]: Deploying daemon osd.5 on smithi178 2023-07-14T17:39:44.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T17:39:44.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:44.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:43 smithi088 ceph-mon[127465]: Deploying daemon osd.5 on smithi178 2023-07-14T17:39:44.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T17:39:44.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:44.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:43 smithi107 ceph-mon[128999]: Deploying daemon osd.5 on smithi178 2023-07-14T17:39:44.986 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:44 smithi178 ceph-mon[129368]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:45.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:44 smithi088 ceph-mon[127465]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:45.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:44 smithi107 ceph-mon[128999]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:46.889 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:46 smithi178 ceph-mon[129368]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:46.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:46 smithi088 ceph-mon[127465]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:46.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:46 smithi107 ceph-mon[128999]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:48.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:48 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:48.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:48 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:48.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:48 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:39:48.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:48 smithi178 ceph-mon[129368]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:49.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:48 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:49.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:48 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:49.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:48 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:39:49.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:48 smithi088 ceph-mon[127465]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:49.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:48 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:49.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:48 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:49.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:48 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:39:49.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:48 smithi107 ceph-mon[128999]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:50.729 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:50.744+0000 7f0afbfff700 1 -- 172.21.15.178:0/3742860083 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0b04060a80 con 0x7f0af00a1150 2023-07-14T17:39:50.730 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 5 on host 'smithi178' 2023-07-14T17:39:50.731 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:50.746+0000 7f0b09697700 1 -- 172.21.15.178:0/3742860083 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0af00a1150 msgr2=0x7f0af00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:39:50.732 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:50.746+0000 7f0b09697700 1 --2- 172.21.15.178:0/3742860083 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0af00a1150 0x7f0af00a3600 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0aec005fd0 tx=0x7f0aec005dd0 comp rx=0 tx=0).stop 2023-07-14T17:39:50.732 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:50.746+0000 7f0b09697700 1 -- 172.21.15.178:0/3742860083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b040ffdb0 msgr2=0x7f0b0419a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:39:50.732 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:50.746+0000 7f0b09697700 1 --2- 172.21.15.178:0/3742860083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b040ffdb0 0x7f0b0419a640 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0af4006010 tx=0x7f0af4004730 comp rx=0 tx=0).stop 2023-07-14T17:39:50.732 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:50.746+0000 7f0b09697700 1 -- 172.21.15.178:0/3742860083 shutdown_connections 2023-07-14T17:39:50.732 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:50.746+0000 7f0b09697700 1 --2- 172.21.15.178:0/3742860083 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b04100740 0x7f0b0419cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:50.732 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:50.746+0000 7f0b09697700 1 --2- 172.21.15.178:0/3742860083 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0af00a1150 0x7f0af00a3600 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:50.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:50.746+0000 7f0b09697700 1 --2- 172.21.15.178:0/3742860083 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b040febb0 0x7f0b04068c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:50.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:50.746+0000 7f0b09697700 1 --2- 172.21.15.178:0/3742860083 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b040ffdb0 0x7f0b0419a640 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:50.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:50.746+0000 7f0b09697700 1 -- 172.21.15.178:0/3742860083 >> 172.21.15.178:0/3742860083 conn(0x7f0b040fa1e0 msgr2=0x7f0b04103380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:39:50.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:50.746+0000 7f0b09697700 1 -- 172.21.15.178:0/3742860083 shutdown_connections 2023-07-14T17:39:50.733 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:39:50.746+0000 7f0b09697700 1 -- 172.21.15.178:0/3742860083 wait complete. 2023-07-14T17:39:51.159 DEBUG:teuthology.orchestra.run.smithi178:osd.5> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.5.service 2023-07-14T17:39:51.161 INFO:tasks.cephadm:Deploying osd.6 on smithi088 with /dev/vg_nvme/lv_2... 2023-07-14T17:39:51.161 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T17:39:51.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:51 smithi178 ceph-mon[129368]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:51.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:51 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:51.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:51 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:51.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:51 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:51.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:51 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:39:51.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:51 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:51.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:51 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:51.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:51 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:51.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:51 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:39:51.205 INFO:journalctl@ceph.osd.5.smithi178.stdout:-- Logs begin at Fri 2023-07-14 17:14:18 UTC. -- 2023-07-14T17:39:51.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:51 smithi088 ceph-mon[127465]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:51.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:51 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:51.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:51 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:51.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:51 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:51.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:51 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:39:51.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:51 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:51.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:51 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:51.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:51 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:51.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:51 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:39:51.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:51 smithi107 ceph-mon[128999]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:51.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:51 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:51.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:51 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:51.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:51 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:51.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:51 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:39:51.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:51 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:51.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:51 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:51.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:51 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:51.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:51 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:39:51.739 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:39:51.916 INFO:journalctl@ceph.osd.5.smithi178.stdout:Jul 14 17:39:51 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-5[140687]: 2023-07-14T17:39:51.778+0000 7f1d10c67640 -1 osd.5 0 log_to_monitors true 2023-07-14T17:39:52.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:52 smithi088 ceph-mon[127465]: from='osd.5 [v2:172.21.15.178:6808/4006439725,v1:172.21.15.178:6809/4006439725]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T17:39:52.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:52 smithi088 ceph-mon[127465]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T17:39:52.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:52 smithi178 ceph-mon[129368]: from='osd.5 [v2:172.21.15.178:6808/4006439725,v1:172.21.15.178:6809/4006439725]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T17:39:52.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:52 smithi178 ceph-mon[129368]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T17:39:52.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:52 smithi107 ceph-mon[128999]: from='osd.5 [v2:172.21.15.178:6808/4006439725,v1:172.21.15.178:6809/4006439725]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T17:39:52.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:52 smithi107 ceph-mon[128999]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T17:39:53.224 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T17:39:53.224 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T17:39:53.224 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2023-07-14T17:39:53.224 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2023-07-14T17:39:53.224 INFO:teuthology.orchestra.run.smithi088.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0107962 s, 971 MB/s 2023-07-14T17:39:53.225 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2023-07-14T17:39:53.339 INFO:journalctl@ceph.osd.5.smithi178.stdout:Jul 14 17:39:53 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-5[140687]: 2023-07-14T17:39:53.063+0000 7f1d062df700 -1 osd.5 0 waiting for initial osdmap 2023-07-14T17:39:53.339 INFO:journalctl@ceph.osd.5.smithi178.stdout:Jul 14 17:39:53 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-5[140687]: 2023-07-14T17:39:53.069+0000 7f1d000d1700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T17:39:53.340 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:53 smithi178 ceph-mon[129368]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:53.340 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:53 smithi178 ceph-mon[129368]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T17:39:53.340 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:53 smithi178 ceph-mon[129368]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T17:39:53.340 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:53 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T17:39:53.340 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:53 smithi178 ceph-mon[129368]: from='osd.5 [v2:172.21.15.178:6808/4006439725,v1:172.21.15.178:6809/4006439725]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:39:53.341 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:53 smithi178 ceph-mon[129368]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:39:53.349 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:53 smithi088 ceph-mon[127465]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:53.350 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:53 smithi088 ceph-mon[127465]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T17:39:53.350 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:53 smithi088 ceph-mon[127465]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T17:39:53.350 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:53 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T17:39:53.350 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:53 smithi088 ceph-mon[127465]: from='osd.5 [v2:172.21.15.178:6808/4006439725,v1:172.21.15.178:6809/4006439725]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:39:53.350 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:53 smithi088 ceph-mon[127465]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:39:53.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:53 smithi107 ceph-mon[128999]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:53.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:53 smithi107 ceph-mon[128999]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T17:39:53.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:53 smithi107 ceph-mon[128999]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T17:39:53.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:53 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T17:39:53.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:53 smithi107 ceph-mon[128999]: from='osd.5 [v2:172.21.15.178:6808/4006439725,v1:172.21.15.178:6809/4006439725]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:39:53.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:53 smithi107 ceph-mon[128999]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:39:53.592 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi088:vg_nvme/lv_2 2023-07-14T17:39:54.154 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:39:54.321 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:54 smithi088 ceph-mon[127465]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2023-07-14T17:39:54.322 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:54 smithi088 ceph-mon[127465]: osdmap e36: 6 total, 5 up, 6 in 2023-07-14T17:39:54.322 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T17:39:54.322 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T17:39:54.322 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:54.322 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:54.322 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:39:54.322 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:39:54.323 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:54.323 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:54.323 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:39:54.323 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:54.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:54 smithi178 ceph-mon[129368]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2023-07-14T17:39:54.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:54 smithi178 ceph-mon[129368]: osdmap e36: 6 total, 5 up, 6 in 2023-07-14T17:39:54.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T17:39:54.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T17:39:54.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:54.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:54.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:39:54.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:39:54.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:54.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:54.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:39:54.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:54.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:54 smithi107 ceph-mon[128999]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2023-07-14T17:39:54.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:54 smithi107 ceph-mon[128999]: osdmap e36: 6 total, 5 up, 6 in 2023-07-14T17:39:54.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T17:39:54.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T17:39:54.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:54.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:54.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:39:54.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:39:54.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:54.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:54.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:39:54.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:39:55.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.373+0000 7fb2aecb0700 1 -- 172.21.15.88:0/1047786448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2a80ffdb0 msgr2=0x7fb2a8100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:39:55.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.373+0000 7fb2aecb0700 1 --2- 172.21.15.88:0/1047786448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2a80ffdb0 0x7fb2a8100200 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb294009990 tx=0x7fb294009ca0 comp rx=0 tx=0).stop 2023-07-14T17:39:55.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.373+0000 7fb2aecb0700 1 -- 172.21.15.88:0/1047786448 shutdown_connections 2023-07-14T17:39:55.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.373+0000 7fb2aecb0700 1 --2- 172.21.15.88:0/1047786448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2a8100740 0x7fb2a8107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:55.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.373+0000 7fb2aecb0700 1 --2- 172.21.15.88:0/1047786448 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2a80febb0 0x7fb2a80fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:55.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.373+0000 7fb2aecb0700 1 --2- 172.21.15.88:0/1047786448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2a80ffdb0 0x7fb2a8100200 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:55.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.373+0000 7fb2aecb0700 1 -- 172.21.15.88:0/1047786448 >> 172.21.15.88:0/1047786448 conn(0x7fb2a80fa1e0 msgr2=0x7fb2a80fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:39:55.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.373+0000 7fb2aecb0700 1 -- 172.21.15.88:0/1047786448 shutdown_connections 2023-07-14T17:39:55.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.373+0000 7fb2aecb0700 1 -- 172.21.15.88:0/1047786448 wait complete. 2023-07-14T17:39:55.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2aecb0700 1 Processor -- start 2023-07-14T17:39:55.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2aecb0700 1 -- start start 2023-07-14T17:39:55.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2aecb0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2a80febb0 0x7fb2a8195880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:39:55.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2aecb0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2a80ffdb0 0x7fb2a8195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:39:55.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2aecb0700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2a8100740 0x7fb2a819ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:39:55.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2aecb0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a819d410 con 0x7fb2a80febb0 2023-07-14T17:39:55.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2aecb0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a819d550 con 0x7fb2a80ffdb0 2023-07-14T17:39:55.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2aecb0700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a819d6e0 con 0x7fb2a8100740 2023-07-14T17:39:55.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2aca4c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2a80febb0 0x7fb2a8195880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:39:55.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2aca4c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2a80febb0 0x7fb2a8195880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40746/0 (socket says 172.21.15.88:40746) 2023-07-14T17:39:55.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2aca4c700 1 -- 172.21.15.88:0/2634555260 learned_addr learned my addr 172.21.15.88:0/2634555260 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:39:55.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2ad24d700 1 --2- 172.21.15.88:0/2634555260 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2a8100740 0x7fb2a819ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:39:55.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2a7fff700 1 --2- 172.21.15.88:0/2634555260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2a80ffdb0 0x7fb2a8195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:39:55.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2aca4c700 1 -- 172.21.15.88:0/2634555260 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2a8100740 msgr2=0x7fb2a819ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:39:55.362 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2aca4c700 1 --2- 172.21.15.88:0/2634555260 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2a8100740 0x7fb2a819ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:55.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2aca4c700 1 -- 172.21.15.88:0/2634555260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2a80ffdb0 msgr2=0x7fb2a8195dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:39:55.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2aca4c700 1 --2- 172.21.15.88:0/2634555260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2a80ffdb0 0x7fb2a8195dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:39:55.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2aca4c700 1 -- 172.21.15.88:0/2634555260 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2940095f0 con 0x7fb2a80febb0 2023-07-14T17:39:55.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.374+0000 7fb2ad24d700 1 --2- 172.21.15.88:0/2634555260 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2a8100740 0x7fb2a819ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:39:55.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.375+0000 7fb2aca4c700 1 --2- 172.21.15.88:0/2634555260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2a80febb0 0x7fb2a8195880 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb29c00c9a0 tx=0x7fb29c00cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:39:55.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.375+0000 7fb2a5ffb700 1 -- 172.21.15.88:0/2634555260 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29c004220 con 0x7fb2a80febb0 2023-07-14T17:39:55.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.375+0000 7fb2a5ffb700 1 -- 172.21.15.88:0/2634555260 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb29c007cf0 con 0x7fb2a80febb0 2023-07-14T17:39:55.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.375+0000 7fb2aecb0700 1 -- 172.21.15.88:0/2634555260 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a819d9c0 con 0x7fb2a80febb0 2023-07-14T17:39:55.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.375+0000 7fb2a5ffb700 1 -- 172.21.15.88:0/2634555260 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29c003c30 con 0x7fb2a80febb0 2023-07-14T17:39:55.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.375+0000 7fb2aecb0700 1 -- 172.21.15.88:0/2634555260 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a819dec0 con 0x7fb2a80febb0 2023-07-14T17:39:55.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.376+0000 7fb2aecb0700 1 -- 172.21.15.88:0/2634555260 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a810a220 con 0x7fb2a80febb0 2023-07-14T17:39:55.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.377+0000 7fb2a5ffb700 1 -- 172.21.15.88:0/2634555260 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb29c004380 con 0x7fb2a80febb0 2023-07-14T17:39:55.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.377+0000 7fb2a5ffb700 1 --2- 172.21.15.88:0/2634555260 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb2980a1150 0x7fb2980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:39:55.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.377+0000 7fb2a7fff700 1 --2- 172.21.15.88:0/2634555260 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb2980a1150 0x7fb2980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:39:55.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.378+0000 7fb2a5ffb700 1 -- 172.21.15.88:0/2634555260 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4562+0+0 (secure 0 0 0) 0x7fb29c0cb750 con 0x7fb2a80febb0 2023-07-14T17:39:55.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.378+0000 7fb2a7fff700 1 --2- 172.21.15.88:0/2634555260 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb2980a1150 0x7fb2980a3600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb294006010 tx=0x7fb294000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:39:55.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.382+0000 7fb2a5ffb700 1 -- 172.21.15.88:0/2634555260 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb29c095ca0 con 0x7fb2a80febb0 2023-07-14T17:39:55.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:55 smithi178 ceph-mon[129368]: purged_snaps scrub starts 2023-07-14T17:39:55.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:55 smithi178 ceph-mon[129368]: purged_snaps scrub ok 2023-07-14T17:39:55.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:55 smithi178 ceph-mon[129368]: Detected new or changed devices on smithi178 2023-07-14T17:39:55.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:55 smithi178 ceph-mon[129368]: Adjusting osd_memory_target on smithi178 to 8584M 2023-07-14T17:39:55.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:55 smithi178 ceph-mon[129368]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:55.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:55 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T17:39:55.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:55 smithi178 ceph-mon[129368]: osd.5 [v2:172.21.15.178:6808/4006439725,v1:172.21.15.178:6809/4006439725] boot 2023-07-14T17:39:55.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:55 smithi178 ceph-mon[129368]: osdmap e37: 6 total, 6 up, 6 in 2023-07-14T17:39:55.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:55 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T17:39:55.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:55 smithi088 ceph-mon[127465]: purged_snaps scrub starts 2023-07-14T17:39:55.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:55 smithi088 ceph-mon[127465]: purged_snaps scrub ok 2023-07-14T17:39:55.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:55 smithi088 ceph-mon[127465]: Detected new or changed devices on smithi178 2023-07-14T17:39:55.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:55 smithi088 ceph-mon[127465]: Adjusting osd_memory_target on smithi178 to 8584M 2023-07-14T17:39:55.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:55 smithi088 ceph-mon[127465]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:55.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:55 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T17:39:55.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:55 smithi088 ceph-mon[127465]: osd.5 [v2:172.21.15.178:6808/4006439725,v1:172.21.15.178:6809/4006439725] boot 2023-07-14T17:39:55.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:55 smithi088 ceph-mon[127465]: osdmap e37: 6 total, 6 up, 6 in 2023-07-14T17:39:55.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:55 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T17:39:55.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:55 smithi107 ceph-mon[128999]: purged_snaps scrub starts 2023-07-14T17:39:55.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:55 smithi107 ceph-mon[128999]: purged_snaps scrub ok 2023-07-14T17:39:55.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:55 smithi107 ceph-mon[128999]: Detected new or changed devices on smithi178 2023-07-14T17:39:55.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:55 smithi107 ceph-mon[128999]: Adjusting osd_memory_target on smithi178 to 8584M 2023-07-14T17:39:55.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:55 smithi107 ceph-mon[128999]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T17:39:55.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:55 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T17:39:55.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:55 smithi107 ceph-mon[128999]: osd.5 [v2:172.21.15.178:6808/4006439725,v1:172.21.15.178:6809/4006439725] boot 2023-07-14T17:39:55.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:55 smithi107 ceph-mon[128999]: osdmap e37: 6 total, 6 up, 6 in 2023-07-14T17:39:55.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:55 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T17:39:55.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:39:55.577+0000 7fb2aecb0700 1 -- 172.21.15.88:0/2634555260 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2a8068890 con 0x7fb2980a1150 2023-07-14T17:39:56.354 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:56 smithi088 ceph-mon[127465]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T17:39:56.354 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:56 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:39:56.354 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:56 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:39:56.355 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:56 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:56.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:56 smithi178 ceph-mon[129368]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T17:39:56.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:56 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:39:56.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:56 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:39:56.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:56 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:56.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:56 smithi107 ceph-mon[128999]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T17:39:56.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:56 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:39:56.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:56 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:39:56.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:56 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:39:57.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:57 smithi178 ceph-mon[129368]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:39:57.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:57 smithi178 ceph-mon[129368]: pgmap v124: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T17:39:57.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:57 smithi178 ceph-mon[129368]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T17:39:57.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:57 smithi088 ceph-mon[127465]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:39:57.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:57 smithi088 ceph-mon[127465]: pgmap v124: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T17:39:57.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:57 smithi088 ceph-mon[127465]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T17:39:57.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:57 smithi107 ceph-mon[128999]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:39:57.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:57 smithi107 ceph-mon[128999]: pgmap v124: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T17:39:57.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:57 smithi107 ceph-mon[128999]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T17:39:58.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:58 smithi178 ceph-mon[129368]: osdmap e40: 6 total, 6 up, 6 in 2023-07-14T17:39:58.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:58 smithi178 ceph-mon[129368]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T17:39:58.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:58 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2650245425' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "206cd369-2456-41c5-8a3f-d1efce5bb45a"}]: dispatch 2023-07-14T17:39:58.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:58 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2650245425' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "206cd369-2456-41c5-8a3f-d1efce5bb45a"}]': finished 2023-07-14T17:39:58.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:58 smithi178 ceph-mon[129368]: osdmap e41: 7 total, 6 up, 7 in 2023-07-14T17:39:58.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:58 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T17:39:58.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:39:58 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/17294600' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:39:58.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:58 smithi088 ceph-mon[127465]: osdmap e40: 6 total, 6 up, 6 in 2023-07-14T17:39:58.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:58 smithi088 ceph-mon[127465]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T17:39:58.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:58 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2650245425' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "206cd369-2456-41c5-8a3f-d1efce5bb45a"}]: dispatch 2023-07-14T17:39:58.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:58 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2650245425' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "206cd369-2456-41c5-8a3f-d1efce5bb45a"}]': finished 2023-07-14T17:39:58.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:58 smithi088 ceph-mon[127465]: osdmap e41: 7 total, 6 up, 7 in 2023-07-14T17:39:58.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:58 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T17:39:58.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:39:58 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/17294600' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:39:58.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:58 smithi107 ceph-mon[128999]: osdmap e40: 6 total, 6 up, 6 in 2023-07-14T17:39:58.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:58 smithi107 ceph-mon[128999]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T17:39:58.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:58 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2650245425' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "206cd369-2456-41c5-8a3f-d1efce5bb45a"}]: dispatch 2023-07-14T17:39:58.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:58 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2650245425' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "206cd369-2456-41c5-8a3f-d1efce5bb45a"}]': finished 2023-07-14T17:39:58.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:58 smithi107 ceph-mon[128999]: osdmap e41: 7 total, 6 up, 7 in 2023-07-14T17:39:58.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:58 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T17:39:58.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:39:58 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/17294600' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:40:01.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:00 smithi178 ceph-mon[129368]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 121 KiB/s, 0 objects/s recovering 2023-07-14T17:40:01.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:00 smithi178 ceph-mon[129368]: overall HEALTH_OK 2023-07-14T17:40:01.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:00 smithi088 ceph-mon[127465]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 121 KiB/s, 0 objects/s recovering 2023-07-14T17:40:01.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:00 smithi088 ceph-mon[127465]: overall HEALTH_OK 2023-07-14T17:40:01.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:00 smithi107 ceph-mon[128999]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 121 KiB/s, 0 objects/s recovering 2023-07-14T17:40:01.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:00 smithi107 ceph-mon[128999]: overall HEALTH_OK 2023-07-14T17:40:02.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:02 smithi088 ceph-mon[127465]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T17:40:03.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:02 smithi178 ceph-mon[129368]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T17:40:03.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:02 smithi107 ceph-mon[128999]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T17:40:05.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:04 smithi178 ceph-mon[129368]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2023-07-14T17:40:05.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:04 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T17:40:05.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:04 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:05.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:04 smithi088 ceph-mon[127465]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2023-07-14T17:40:05.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:04 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T17:40:05.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:04 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:05.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:04 smithi107 ceph-mon[128999]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 74 KiB/s, 0 objects/s recovering 2023-07-14T17:40:05.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:04 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T17:40:05.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:04 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:06.101 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:05 smithi088 ceph-mon[127465]: Deploying daemon osd.6 on smithi088 2023-07-14T17:40:06.143 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:05 smithi107 ceph-mon[128999]: Deploying daemon osd.6 on smithi088 2023-07-14T17:40:06.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:05 smithi178 ceph-mon[129368]: Deploying daemon osd.6 on smithi088 2023-07-14T17:40:07.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:06 smithi178 ceph-mon[129368]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-07-14T17:40:07.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:06 smithi088 ceph-mon[127465]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-07-14T17:40:07.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:06 smithi107 ceph-mon[128999]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-07-14T17:40:09.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:08 smithi178 ceph-mon[129368]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T17:40:09.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:08 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:09.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:08 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:09.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:08 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:40:09.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:08 smithi088 ceph-mon[127465]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T17:40:09.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:08 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:09.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:08 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:09.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:08 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:40:09.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:08 smithi107 ceph-mon[128999]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T17:40:09.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:08 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:09.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:08 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:09.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:08 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:40:10.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:10 smithi088 ceph-mon[127465]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T17:40:11.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:10 smithi178 ceph-mon[129368]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T17:40:11.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:10 smithi107 ceph-mon[128999]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T17:40:11.611 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 6 on host 'smithi088' 2023-07-14T17:40:11.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:11.625+0000 7fb2a5ffb700 1 -- 172.21.15.88:0/2634555260 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb2a8068890 con 0x7fb2980a1150 2023-07-14T17:40:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:11.628+0000 7fb2aecb0700 1 -- 172.21.15.88:0/2634555260 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb2980a1150 msgr2=0x7fb2980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:40:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:11.628+0000 7fb2aecb0700 1 --2- 172.21.15.88:0/2634555260 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb2980a1150 0x7fb2980a3600 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb294006010 tx=0x7fb294000bc0 comp rx=0 tx=0).stop 2023-07-14T17:40:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:11.628+0000 7fb2aecb0700 1 -- 172.21.15.88:0/2634555260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2a80febb0 msgr2=0x7fb2a8195880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:40:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:11.628+0000 7fb2aecb0700 1 --2- 172.21.15.88:0/2634555260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2a80febb0 0x7fb2a8195880 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb29c00c9a0 tx=0x7fb29c00cd60 comp rx=0 tx=0).stop 2023-07-14T17:40:11.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:11.628+0000 7fb2aecb0700 1 -- 172.21.15.88:0/2634555260 shutdown_connections 2023-07-14T17:40:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:11.628+0000 7fb2aecb0700 1 --2- 172.21.15.88:0/2634555260 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb2a8100740 0x7fb2a819ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:11.628+0000 7fb2aecb0700 1 --2- 172.21.15.88:0/2634555260 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb2980a1150 0x7fb2980a3600 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:11.628+0000 7fb2aecb0700 1 --2- 172.21.15.88:0/2634555260 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2a80febb0 0x7fb2a8195880 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:11.628+0000 7fb2aecb0700 1 --2- 172.21.15.88:0/2634555260 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2a80ffdb0 0x7fb2a8195dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:11.628+0000 7fb2aecb0700 1 -- 172.21.15.88:0/2634555260 >> 172.21.15.88:0/2634555260 conn(0x7fb2a80fa1e0 msgr2=0x7fb2a8068180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:40:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:11.628+0000 7fb2aecb0700 1 -- 172.21.15.88:0/2634555260 shutdown_connections 2023-07-14T17:40:11.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:11.628+0000 7fb2aecb0700 1 -- 172.21.15.88:0/2634555260 wait complete. 2023-07-14T17:40:12.043 DEBUG:teuthology.orchestra.run.smithi088:osd.6> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.6.service 2023-07-14T17:40:12.045 INFO:tasks.cephadm:Deploying osd.7 on smithi107 with /dev/vg_nvme/lv_2... 2023-07-14T17:40:12.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T17:40:12.075 INFO:journalctl@ceph.osd.6.smithi088.stdout:-- Logs begin at Fri 2023-07-14 17:14:35 UTC. -- 2023-07-14T17:40:12.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:12 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:12.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:12 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:12.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:12 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:12.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:12 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:12.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:12 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:12.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:12 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:12.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:12 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:12.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:12 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:12.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:12 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:12.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:12 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:12.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:12 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:12.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:12 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:12.624 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T17:40:12.895 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:40:12 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:40:12.741+0000 7f74fe981640 -1 osd.6 0 log_to_monitors true 2023-07-14T17:40:13.114 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:13 smithi107 ceph-mon[128999]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T17:40:13.114 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:13 smithi107 ceph-mon[128999]: from='osd.6 [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T17:40:13.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:13 smithi088 ceph-mon[127465]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T17:40:13.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:13 smithi088 ceph-mon[127465]: from='osd.6 [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T17:40:13.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:13 smithi178 ceph-mon[129368]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T17:40:13.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:13 smithi178 ceph-mon[129368]: from='osd.6 [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T17:40:14.103 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T17:40:14.103 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T17:40:14.103 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2023-07-14T17:40:14.104 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2023-07-14T17:40:14.104 INFO:teuthology.orchestra.run.smithi107.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126588 s, 828 MB/s 2023-07-14T17:40:14.104 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 2023-07-14T17:40:14.105 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2023-07-14T17:40:14.578 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi107:vg_nvme/lv_2 2023-07-14T17:40:14.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:14 smithi178 ceph-mon[129368]: from='osd.6 [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T17:40:14.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:14 smithi178 ceph-mon[129368]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T17:40:14.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T17:40:14.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:14 smithi178 ceph-mon[129368]: from='osd.6 [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T17:40:14.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:14 smithi178 ceph-mon[129368]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T17:40:14.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:14.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:14.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:14.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:14.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:14.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:14.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:14.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:40:14.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:14.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:40:14.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:14.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:40:14.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:14.943 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:40:14.616+0000 7f74f3ff9700 -1 osd.6 0 waiting for initial osdmap 2023-07-14T17:40:14.943 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:40:14.623+0000 7f74ed5ea700 -1 osd.6 43 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T17:40:14.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-mon[127465]: from='osd.6 [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T17:40:14.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-mon[127465]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T17:40:14.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T17:40:14.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-mon[127465]: from='osd.6 [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T17:40:14.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-mon[127465]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T17:40:14.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:14.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:14.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:14.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:14.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:14.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:14.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:14.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:40:14.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:14.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:40:14.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:14.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:40:14.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:14.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:14 smithi107 ceph-mon[128999]: from='osd.6 [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T17:40:14.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:14 smithi107 ceph-mon[128999]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T17:40:14.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T17:40:14.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:14 smithi107 ceph-mon[128999]: from='osd.6 [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T17:40:14.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:14 smithi107 ceph-mon[128999]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T17:40:14.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:14.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:14.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:14.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:14.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:14.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:14.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:14.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:40:14.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:14.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:40:14.978 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:14.979 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:40:14.979 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:15.127 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T17:40:15.902 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:15 smithi107 ceph-mon[128999]: Detected new or changed devices on smithi088 2023-07-14T17:40:15.903 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:15 smithi107 ceph-mon[128999]: Adjusting osd_memory_target on smithi088 to 5722M 2023-07-14T17:40:15.903 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:15 smithi107 ceph-mon[128999]: from='osd.6 [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T17:40:15.903 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:15 smithi107 ceph-mon[128999]: osdmap e43: 7 total, 6 up, 7 in 2023-07-14T17:40:15.903 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:15 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T17:40:15.903 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:15 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T17:40:15.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:15 smithi178 ceph-mon[129368]: Detected new or changed devices on smithi088 2023-07-14T17:40:15.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:15 smithi178 ceph-mon[129368]: Adjusting osd_memory_target on smithi088 to 5722M 2023-07-14T17:40:15.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:15 smithi178 ceph-mon[129368]: from='osd.6 [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T17:40:15.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:15 smithi178 ceph-mon[129368]: osdmap e43: 7 total, 6 up, 7 in 2023-07-14T17:40:15.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:15 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T17:40:15.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:15 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T17:40:15.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:15 smithi088 ceph-mon[127465]: Detected new or changed devices on smithi088 2023-07-14T17:40:15.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:15 smithi088 ceph-mon[127465]: Adjusting osd_memory_target on smithi088 to 5722M 2023-07-14T17:40:15.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:15 smithi088 ceph-mon[127465]: from='osd.6 [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T17:40:15.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:15 smithi088 ceph-mon[127465]: osdmap e43: 7 total, 6 up, 7 in 2023-07-14T17:40:15.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:15 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T17:40:15.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:15 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T17:40:16.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.344+0000 7fe495b2d700 1 -- 172.21.15.107:0/2581331078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4900ffe70 msgr2=0x7fe4901002e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:40:16.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.344+0000 7fe495b2d700 1 --2- 172.21.15.107:0/2581331078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4900ffe70 0x7fe4901002e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe484009990 tx=0x7fe484009ca0 comp rx=0 tx=0).stop 2023-07-14T17:40:16.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.344+0000 7fe495b2d700 1 -- 172.21.15.107:0/2581331078 shutdown_connections 2023-07-14T17:40:16.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.344+0000 7fe495b2d700 1 --2- 172.21.15.107:0/2581331078 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe490100820 0x7fe4901070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:16.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.344+0000 7fe495b2d700 1 --2- 172.21.15.107:0/2581331078 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4900febd0 0x7fe4900fefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:16.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.344+0000 7fe495b2d700 1 --2- 172.21.15.107:0/2581331078 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4900ffe70 0x7fe4901002e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:16.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.344+0000 7fe495b2d700 1 -- 172.21.15.107:0/2581331078 >> 172.21.15.107:0/2581331078 conn(0x7fe4900fa240 msgr2=0x7fe4900fc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:40:16.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.344+0000 7fe495b2d700 1 -- 172.21.15.107:0/2581331078 shutdown_connections 2023-07-14T17:40:16.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.344+0000 7fe495b2d700 1 -- 172.21.15.107:0/2581331078 wait complete. 2023-07-14T17:40:16.331 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.345+0000 7fe495b2d700 1 Processor -- start 2023-07-14T17:40:16.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.345+0000 7fe495b2d700 1 -- start start 2023-07-14T17:40:16.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe495b2d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4900febd0 0x7fe4900695e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:40:16.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe495b2d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4900ffe70 0x7fe49019a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:40:16.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe495b2d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe490100820 0x7fe49019cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:40:16.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe495b2d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49019d3b0 con 0x7fe4900febd0 2023-07-14T17:40:16.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe495b2d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49019d520 con 0x7fe4900ffe70 2023-07-14T17:40:16.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe495b2d700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49019d6e0 con 0x7fe490100820 2023-07-14T17:40:16.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe48effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4900ffe70 0x7fe49019a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:40:16.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe48effd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4900ffe70 0x7fe49019a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42626/0 (socket says 172.21.15.107:42626) 2023-07-14T17:40:16.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe48effd700 1 -- 172.21.15.107:0/2852264650 learned_addr learned my addr 172.21.15.107:0/2852264650 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-07-14T17:40:16.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe48f7fe700 1 --2- 172.21.15.107:0/2852264650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4900febd0 0x7fe4900695e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:40:16.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe48ffff700 1 --2- 172.21.15.107:0/2852264650 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe490100820 0x7fe49019cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:40:16.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe48effd700 1 -- 172.21.15.107:0/2852264650 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe490100820 msgr2=0x7fe49019cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:40:16.335 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe48effd700 1 --2- 172.21.15.107:0/2852264650 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe490100820 0x7fe49019cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:16.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe48effd700 1 -- 172.21.15.107:0/2852264650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4900febd0 msgr2=0x7fe4900695e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:40:16.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe48effd700 1 --2- 172.21.15.107:0/2852264650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4900febd0 0x7fe4900695e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:16.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe48effd700 1 -- 172.21.15.107:0/2852264650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4840095f0 con 0x7fe4900ffe70 2023-07-14T17:40:16.336 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe48effd700 1 --2- 172.21.15.107:0/2852264650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4900ffe70 0x7fe49019a580 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe484006010 tx=0x7fe484003b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:40:16.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe48cff9700 1 -- 172.21.15.107:0/2852264650 <== mon.2 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48401d070 con 0x7fe4900ffe70 2023-07-14T17:40:16.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe495b2d700 1 -- 172.21.15.107:0/2852264650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe49019d960 con 0x7fe4900ffe70 2023-07-14T17:40:16.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe48cff9700 1 -- 172.21.15.107:0/2852264650 <== mon.2 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe48400bd70 con 0x7fe4900ffe70 2023-07-14T17:40:16.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.346+0000 7fe48cff9700 1 -- 172.21.15.107:0/2852264650 <== mon.2 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484022e30 con 0x7fe4900ffe70 2023-07-14T17:40:16.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.347+0000 7fe495b2d700 1 -- 172.21.15.107:0/2852264650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe49019dd80 con 0x7fe4900ffe70 2023-07-14T17:40:16.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.347+0000 7fe495b2d700 1 -- 172.21.15.107:0/2852264650 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe490106220 con 0x7fe4900ffe70 2023-07-14T17:40:16.337 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.348+0000 7fe48cff9700 1 -- 172.21.15.107:0/2852264650 <== mon.2 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe484003ff0 con 0x7fe4900ffe70 2023-07-14T17:40:16.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.349+0000 7fe48cff9700 1 --2- 172.21.15.107:0/2852264650 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe4780a1150 0x7fe4780a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:40:16.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.349+0000 7fe48cff9700 1 -- 172.21.15.107:0/2852264650 <== mon.2 v2:172.21.15.107:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4965+0+0 (secure 0 0 0) 0x7fe4840cd890 con 0x7fe4900ffe70 2023-07-14T17:40:16.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.349+0000 7fe48f7fe700 1 --2- 172.21.15.107:0/2852264650 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe4780a1150 0x7fe4780a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:40:16.338 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.349+0000 7fe48f7fe700 1 --2- 172.21.15.107:0/2852264650 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe4780a1150 0x7fe4780a3600 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe480005fd0 tx=0x7fe48000a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:40:16.339 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.352+0000 7fe48cff9700 1 -- 172.21.15.107:0/2852264650 <== mon.2 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe484097d80 con 0x7fe4900ffe70 2023-07-14T17:40:16.528 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:16.543+0000 7fe495b2d700 1 -- 172.21.15.107:0/2852264650 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe490060a80 con 0x7fe4780a1150 2023-07-14T17:40:16.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:16 smithi178 ceph-mon[129368]: purged_snaps scrub starts 2023-07-14T17:40:16.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:16 smithi178 ceph-mon[129368]: purged_snaps scrub ok 2023-07-14T17:40:16.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:16 smithi178 ceph-mon[129368]: osd.6 [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128] boot 2023-07-14T17:40:16.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:16 smithi178 ceph-mon[129368]: osdmap e44: 7 total, 7 up, 7 in 2023-07-14T17:40:16.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:16 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T17:40:16.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:16 smithi178 ceph-mon[129368]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:16.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:16 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:40:16.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:16 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:40:16.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:16 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:16.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:16 smithi088 ceph-mon[127465]: purged_snaps scrub starts 2023-07-14T17:40:16.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:16 smithi088 ceph-mon[127465]: purged_snaps scrub ok 2023-07-14T17:40:16.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:16 smithi088 ceph-mon[127465]: osd.6 [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128] boot 2023-07-14T17:40:16.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:16 smithi088 ceph-mon[127465]: osdmap e44: 7 total, 7 up, 7 in 2023-07-14T17:40:16.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:16 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T17:40:16.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:16 smithi088 ceph-mon[127465]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:16.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:16 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:40:16.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:16 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:40:16.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:16 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:16.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:16 smithi107 ceph-mon[128999]: purged_snaps scrub starts 2023-07-14T17:40:16.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:16 smithi107 ceph-mon[128999]: purged_snaps scrub ok 2023-07-14T17:40:16.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:16 smithi107 ceph-mon[128999]: osd.6 [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128] boot 2023-07-14T17:40:16.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:16 smithi107 ceph-mon[128999]: osdmap e44: 7 total, 7 up, 7 in 2023-07-14T17:40:16.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:16 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T17:40:16.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:16 smithi107 ceph-mon[128999]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:16.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:16 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:40:16.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:16 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:40:16.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:16 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:17.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:17 smithi178 ceph-mon[129368]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:40:17.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:17 smithi178 ceph-mon[129368]: osdmap e45: 7 total, 7 up, 7 in 2023-07-14T17:40:17.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:17 smithi088 ceph-mon[127465]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:40:17.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:17 smithi088 ceph-mon[127465]: osdmap e45: 7 total, 7 up, 7 in 2023-07-14T17:40:17.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:17 smithi107 ceph-mon[128999]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:40:17.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:17 smithi107 ceph-mon[128999]: osdmap e45: 7 total, 7 up, 7 in 2023-07-14T17:40:18.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:18 smithi178 ceph-mon[129368]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:18.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:18 smithi088 ceph-mon[127465]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:18.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:18 smithi107 ceph-mon[128999]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:19.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:19 smithi178 ceph-mon[129368]: from='client.? 172.21.15.107:0/2276984210' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59b9919f-52fb-42b6-b1d4-dee1df75c23a"}]: dispatch 2023-07-14T17:40:19.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:19 smithi178 ceph-mon[129368]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59b9919f-52fb-42b6-b1d4-dee1df75c23a"}]: dispatch 2023-07-14T17:40:19.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:19 smithi178 ceph-mon[129368]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59b9919f-52fb-42b6-b1d4-dee1df75c23a"}]': finished 2023-07-14T17:40:19.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:19 smithi178 ceph-mon[129368]: osdmap e46: 8 total, 7 up, 8 in 2023-07-14T17:40:19.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:19 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T17:40:19.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:19 smithi178 ceph-mon[129368]: from='client.? 172.21.15.107:0/5334225' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:40:19.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:19 smithi088 ceph-mon[127465]: from='client.? 172.21.15.107:0/2276984210' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59b9919f-52fb-42b6-b1d4-dee1df75c23a"}]: dispatch 2023-07-14T17:40:19.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:19 smithi088 ceph-mon[127465]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59b9919f-52fb-42b6-b1d4-dee1df75c23a"}]: dispatch 2023-07-14T17:40:19.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:19 smithi088 ceph-mon[127465]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59b9919f-52fb-42b6-b1d4-dee1df75c23a"}]': finished 2023-07-14T17:40:19.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:19 smithi088 ceph-mon[127465]: osdmap e46: 8 total, 7 up, 8 in 2023-07-14T17:40:19.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:19 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T17:40:19.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:19 smithi088 ceph-mon[127465]: from='client.? 172.21.15.107:0/5334225' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:40:19.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:19 smithi107 ceph-mon[128999]: from='client.? 172.21.15.107:0/2276984210' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59b9919f-52fb-42b6-b1d4-dee1df75c23a"}]: dispatch 2023-07-14T17:40:19.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:19 smithi107 ceph-mon[128999]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "59b9919f-52fb-42b6-b1d4-dee1df75c23a"}]: dispatch 2023-07-14T17:40:19.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:19 smithi107 ceph-mon[128999]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "59b9919f-52fb-42b6-b1d4-dee1df75c23a"}]': finished 2023-07-14T17:40:19.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:19 smithi107 ceph-mon[128999]: osdmap e46: 8 total, 7 up, 8 in 2023-07-14T17:40:19.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:19 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T17:40:19.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:19 smithi107 ceph-mon[128999]: from='client.? 172.21.15.107:0/5334225' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:40:20.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:20 smithi178 ceph-mon[129368]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:20.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:20 smithi088 ceph-mon[127465]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:20.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:20 smithi107 ceph-mon[128999]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:22.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:22 smithi107 ceph-mon[128999]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:23.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:22 smithi178 ceph-mon[129368]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:23.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:22 smithi088 ceph-mon[127465]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:25.138 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:24 smithi107 ceph-mon[128999]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:25.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:24 smithi178 ceph-mon[129368]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:25.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:24 smithi088 ceph-mon[127465]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:26.133 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:25 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T17:40:26.133 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:25 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:26.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T17:40:26.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:26.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T17:40:26.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:26.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:26 smithi107 ceph-mon[128999]: Deploying daemon osd.7 on smithi107 2023-07-14T17:40:26.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:26 smithi107 ceph-mon[128999]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:27.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:26 smithi178 ceph-mon[129368]: Deploying daemon osd.7 on smithi107 2023-07-14T17:40:27.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:26 smithi178 ceph-mon[129368]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:27.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:26 smithi088 ceph-mon[127465]: Deploying daemon osd.7 on smithi107 2023-07-14T17:40:27.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:26 smithi088 ceph-mon[127465]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:29.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:28 smithi178 ceph-mon[129368]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:29.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:28 smithi088 ceph-mon[127465]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:29.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:28 smithi107 ceph-mon[128999]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:30.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:30 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:30.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:30 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:30.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:30 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:40:30.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:30 smithi107 ceph-mon[128999]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:31.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:30 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:31.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:30 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:31.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:30 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:40:31.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:30 smithi178 ceph-mon[129368]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:31.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:30 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:31.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:30 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:31.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:30 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:40:31.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:30 smithi088 ceph-mon[127465]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:32.739 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:32.754+0000 7fe48cff9700 1 -- 172.21.15.107:0/2852264650 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe490060a80 con 0x7fe4780a1150 2023-07-14T17:40:32.739 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 7 on host 'smithi107' 2023-07-14T17:40:32.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:32.757+0000 7fe495b2d700 1 -- 172.21.15.107:0/2852264650 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe4780a1150 msgr2=0x7fe4780a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:40:32.742 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:32.758+0000 7fe495b2d700 1 --2- 172.21.15.107:0/2852264650 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe4780a1150 0x7fe4780a3600 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe480005fd0 tx=0x7fe48000a560 comp rx=0 tx=0).stop 2023-07-14T17:40:32.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:32.758+0000 7fe495b2d700 1 -- 172.21.15.107:0/2852264650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4900ffe70 msgr2=0x7fe49019a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:40:32.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:32.758+0000 7fe495b2d700 1 --2- 172.21.15.107:0/2852264650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4900ffe70 0x7fe49019a580 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe484006010 tx=0x7fe484003b00 comp rx=0 tx=0).stop 2023-07-14T17:40:32.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:32.758+0000 7fe495b2d700 1 -- 172.21.15.107:0/2852264650 shutdown_connections 2023-07-14T17:40:32.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:32.758+0000 7fe495b2d700 1 --2- 172.21.15.107:0/2852264650 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe490100820 0x7fe49019cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:32.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:32.758+0000 7fe495b2d700 1 --2- 172.21.15.107:0/2852264650 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe4780a1150 0x7fe4780a3600 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:32.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:32.758+0000 7fe495b2d700 1 --2- 172.21.15.107:0/2852264650 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4900febd0 0x7fe4900695e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:32.743 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:32.758+0000 7fe495b2d700 1 --2- 172.21.15.107:0/2852264650 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe4900ffe70 0x7fe49019a580 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:32.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:32.758+0000 7fe495b2d700 1 -- 172.21.15.107:0/2852264650 >> 172.21.15.107:0/2852264650 conn(0x7fe4900fa240 msgr2=0x7fe490103360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:40:32.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:32.758+0000 7fe495b2d700 1 -- 172.21.15.107:0/2852264650 shutdown_connections 2023-07-14T17:40:32.744 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:40:32.758+0000 7fe495b2d700 1 -- 172.21.15.107:0/2852264650 wait complete. 2023-07-14T17:40:32.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:32 smithi107 ceph-mon[128999]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:32.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:32.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:32.918 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:32.918 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:33.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:32 smithi178 ceph-mon[129368]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:33.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:33.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:33.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:33.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:33.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:32 smithi088 ceph-mon[127465]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:33.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:33.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:33.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:33.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:33.205 DEBUG:teuthology.orchestra.run.smithi107:osd.7> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.7.service 2023-07-14T17:40:33.207 INFO:tasks.cephadm:Deploying osd.8 on smithi178 with /dev/vg_nvme/lv_2... 2023-07-14T17:40:33.207 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T17:40:33.242 INFO:journalctl@ceph.osd.7.smithi107.stdout:-- Logs begin at Fri 2023-07-14 17:14:00 UTC. -- 2023-07-14T17:40:33.771 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T17:40:33.959 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:40:33 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:40:33.774+0000 7f5afca8e640 -1 osd.7 0 log_to_monitors true 2023-07-14T17:40:33.959 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:33 smithi107 ceph-mon[128999]: from='osd.7 [v2:172.21.15.107:6816/2214437741,v1:172.21.15.107:6817/2214437741]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T17:40:33.959 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:33 smithi107 ceph-mon[128999]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T17:40:34.158 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:33 smithi178 ceph-mon[129368]: from='osd.7 [v2:172.21.15.107:6816/2214437741,v1:172.21.15.107:6817/2214437741]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T17:40:34.158 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:33 smithi178 ceph-mon[129368]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T17:40:34.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:33 smithi088 ceph-mon[127465]: from='osd.7 [v2:172.21.15.107:6816/2214437741,v1:172.21.15.107:6817/2214437741]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T17:40:34.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:33 smithi088 ceph-mon[127465]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T17:40:35.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:34 smithi178 ceph-mon[129368]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:35.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:34 smithi178 ceph-mon[129368]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T17:40:35.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:34 smithi178 ceph-mon[129368]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T17:40:35.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T17:40:35.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:34 smithi178 ceph-mon[129368]: from='osd.7 [v2:172.21.15.107:6816/2214437741,v1:172.21.15.107:6817/2214437741]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:40:35.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:34 smithi178 ceph-mon[129368]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:40:35.192 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:40:34 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:40:34.910+0000 7f5af2106700 -1 osd.7 0 waiting for initial osdmap 2023-07-14T17:40:35.193 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:40:34 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:40:34.919+0000 7f5aeb6f7700 -1 osd.7 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T17:40:35.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:34 smithi107 ceph-mon[128999]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:35.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:34 smithi107 ceph-mon[128999]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T17:40:35.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:34 smithi107 ceph-mon[128999]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T17:40:35.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:34 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T17:40:35.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:34 smithi107 ceph-mon[128999]: from='osd.7 [v2:172.21.15.107:6816/2214437741,v1:172.21.15.107:6817/2214437741]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:40:35.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:34 smithi107 ceph-mon[128999]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:40:35.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:34 smithi088 ceph-mon[127465]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:35.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:34 smithi088 ceph-mon[127465]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T17:40:35.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:34 smithi088 ceph-mon[127465]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T17:40:35.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T17:40:35.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:34 smithi088 ceph-mon[127465]: from='osd.7 [v2:172.21.15.107:6816/2214437741,v1:172.21.15.107:6817/2214437741]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:40:35.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:34 smithi088 ceph-mon[127465]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:40:35.250 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T17:40:35.250 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T17:40:35.250 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2023-07-14T17:40:35.250 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2023-07-14T17:40:35.250 INFO:teuthology.orchestra.run.smithi178.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0082378 s, 1.3 GB/s 2023-07-14T17:40:35.251 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2023-07-14T17:40:35.714 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi178:vg_nvme/lv_2 2023-07-14T17:40:36.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2023-07-14T17:40:36.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: osdmap e48: 8 total, 7 up, 8 in 2023-07-14T17:40:36.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T17:40:36.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T17:40:36.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:36.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:36.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:36.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:36.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:36.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:36.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:36.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:40:36.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:36.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: osd.7 [v2:172.21.15.107:6816/2214437741,v1:172.21.15.107:6817/2214437741] boot 2023-07-14T17:40:36.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: osdmap e49: 8 total, 8 up, 8 in 2023-07-14T17:40:36.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T17:40:36.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:40:36.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:36.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:40:36.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:35 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:36.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2023-07-14T17:40:36.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: osdmap e48: 8 total, 7 up, 8 in 2023-07-14T17:40:36.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T17:40:36.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T17:40:36.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:36.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:36.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:36.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:36.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:36.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:36.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:36.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:40:36.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:36.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: osd.7 [v2:172.21.15.107:6816/2214437741,v1:172.21.15.107:6817/2214437741] boot 2023-07-14T17:40:36.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: osdmap e49: 8 total, 8 up, 8 in 2023-07-14T17:40:36.195 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T17:40:36.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:40:36.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:36.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:40:36.196 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:35 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:36.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2023-07-14T17:40:36.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: osdmap e48: 8 total, 7 up, 8 in 2023-07-14T17:40:36.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T17:40:36.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T17:40:36.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:36.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:36.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:36.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:36.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:36.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:36.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:36.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:40:36.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:36.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: osd.7 [v2:172.21.15.107:6816/2214437741,v1:172.21.15.107:6817/2214437741] boot 2023-07-14T17:40:36.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: osdmap e49: 8 total, 8 up, 8 in 2023-07-14T17:40:36.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T17:40:36.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:40:36.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:36.228 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:40:36.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:35 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:36.268 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T17:40:37.135 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:36 smithi178 ceph-mon[129368]: purged_snaps scrub starts 2023-07-14T17:40:37.135 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:36 smithi178 ceph-mon[129368]: purged_snaps scrub ok 2023-07-14T17:40:37.136 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:36 smithi178 ceph-mon[129368]: Detected new or changed devices on smithi107 2023-07-14T17:40:37.136 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:36 smithi178 ceph-mon[129368]: Adjusting osd_memory_target on smithi107 to 5722M 2023-07-14T17:40:37.136 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:36 smithi178 ceph-mon[129368]: pgmap v155: 1 pgs: 1 unknown; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:37.137 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:36 smithi178 ceph-mon[129368]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T17:40:37.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:36 smithi088 ceph-mon[127465]: purged_snaps scrub starts 2023-07-14T17:40:37.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:36 smithi088 ceph-mon[127465]: purged_snaps scrub ok 2023-07-14T17:40:37.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:36 smithi088 ceph-mon[127465]: Detected new or changed devices on smithi107 2023-07-14T17:40:37.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:36 smithi088 ceph-mon[127465]: Adjusting osd_memory_target on smithi107 to 5722M 2023-07-14T17:40:37.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:36 smithi088 ceph-mon[127465]: pgmap v155: 1 pgs: 1 unknown; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:37.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:36 smithi088 ceph-mon[127465]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T17:40:37.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:36 smithi107 ceph-mon[128999]: purged_snaps scrub starts 2023-07-14T17:40:37.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:36 smithi107 ceph-mon[128999]: purged_snaps scrub ok 2023-07-14T17:40:37.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:36 smithi107 ceph-mon[128999]: Detected new or changed devices on smithi107 2023-07-14T17:40:37.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:36 smithi107 ceph-mon[128999]: Adjusting osd_memory_target on smithi107 to 5722M 2023-07-14T17:40:37.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:36 smithi107 ceph-mon[128999]: pgmap v155: 1 pgs: 1 unknown; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T17:40:37.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:36 smithi107 ceph-mon[128999]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T17:40:37.579 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.595+0000 7fe53218d700 1 -- 172.21.15.178:0/2258769523 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe52c0febd0 msgr2=0x7fe52c0fefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:40:37.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.595+0000 7fe53218d700 1 --2- 172.21.15.178:0/2258769523 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe52c0febd0 0x7fe52c0fefe0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fe51c009990 tx=0x7fe51c009ca0 comp rx=0 tx=0).stop 2023-07-14T17:40:37.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.595+0000 7fe53218d700 1 -- 172.21.15.178:0/2258769523 shutdown_connections 2023-07-14T17:40:37.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.595+0000 7fe53218d700 1 --2- 172.21.15.178:0/2258769523 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe52c100820 0x7fe52c1070c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:37.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.595+0000 7fe53218d700 1 --2- 172.21.15.178:0/2258769523 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe52c0febd0 0x7fe52c0fefe0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:37.580 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.595+0000 7fe53218d700 1 --2- 172.21.15.178:0/2258769523 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe52c0ffe70 0x7fe52c1002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:37.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.595+0000 7fe53218d700 1 -- 172.21.15.178:0/2258769523 >> 172.21.15.178:0/2258769523 conn(0x7fe52c0fa240 msgr2=0x7fe52c0fc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:40:37.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.596+0000 7fe53218d700 1 -- 172.21.15.178:0/2258769523 shutdown_connections 2023-07-14T17:40:37.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.596+0000 7fe53218d700 1 -- 172.21.15.178:0/2258769523 wait complete. 2023-07-14T17:40:37.581 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.596+0000 7fe53218d700 1 Processor -- start 2023-07-14T17:40:37.582 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.596+0000 7fe53218d700 1 -- start start 2023-07-14T17:40:37.583 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.596+0000 7fe53218d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe52c0febd0 0x7fe52c1947b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:40:37.583 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.596+0000 7fe53218d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe52c0ffe70 0x7fe52c194cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:40:37.583 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.596+0000 7fe53218d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe52c100820 0x7fe52c19bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:40:37.583 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.596+0000 7fe53218d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c19c340 con 0x7fe52c0febd0 2023-07-14T17:40:37.583 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.596+0000 7fe53218d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c19c480 con 0x7fe52c100820 2023-07-14T17:40:37.583 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.596+0000 7fe53218d700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52c19c610 con 0x7fe52c0ffe70 2023-07-14T17:40:37.584 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.597+0000 7fe52affd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe52c0ffe70 0x7fe52c194cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:40:37.584 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.597+0000 7fe52affd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe52c0ffe70 0x7fe52c194cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:54852/0 (socket says 172.21.15.178:54852) 2023-07-14T17:40:37.584 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.597+0000 7fe52affd700 1 -- 172.21.15.178:0/3510455655 learned_addr learned my addr 172.21.15.178:0/3510455655 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-07-14T17:40:37.584 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.597+0000 7fe52b7fe700 1 --2- 172.21.15.178:0/3510455655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe52c0febd0 0x7fe52c1947b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:40:37.584 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.597+0000 7fe52bfff700 1 --2- 172.21.15.178:0/3510455655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe52c100820 0x7fe52c19bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:40:37.584 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.597+0000 7fe52affd700 1 -- 172.21.15.178:0/3510455655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe52c100820 msgr2=0x7fe52c19bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:40:37.584 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.597+0000 7fe52affd700 1 --2- 172.21.15.178:0/3510455655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe52c100820 0x7fe52c19bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:37.584 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.597+0000 7fe52affd700 1 -- 172.21.15.178:0/3510455655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe52c0febd0 msgr2=0x7fe52c1947b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:40:37.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.597+0000 7fe52affd700 1 --2- 172.21.15.178:0/3510455655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe52c0febd0 0x7fe52c1947b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:37.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.597+0000 7fe52affd700 1 -- 172.21.15.178:0/3510455655 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51c0095f0 con 0x7fe52c0ffe70 2023-07-14T17:40:37.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.597+0000 7fe52affd700 1 --2- 172.21.15.178:0/3510455655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe52c0ffe70 0x7fe52c194cf0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe52000d980 tx=0x7fe52000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:40:37.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.597+0000 7fe528ff9700 1 -- 172.21.15.178:0/3510455655 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5200098d0 con 0x7fe52c0ffe70 2023-07-14T17:40:37.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.597+0000 7fe528ff9700 1 -- 172.21.15.178:0/3510455655 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe520010460 con 0x7fe52c0ffe70 2023-07-14T17:40:37.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.597+0000 7fe53218d700 1 -- 172.21.15.178:0/3510455655 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52c068b40 con 0x7fe52c0ffe70 2023-07-14T17:40:37.585 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.597+0000 7fe528ff9700 1 -- 172.21.15.178:0/3510455655 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52000f620 con 0x7fe52c0ffe70 2023-07-14T17:40:37.586 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.597+0000 7fe53218d700 1 -- 172.21.15.178:0/3510455655 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52c069040 con 0x7fe52c0ffe70 2023-07-14T17:40:37.586 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.598+0000 7fe53218d700 1 -- 172.21.15.178:0/3510455655 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe52c10a250 con 0x7fe52c0ffe70 2023-07-14T17:40:37.586 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.599+0000 7fe528ff9700 1 -- 172.21.15.178:0/3510455655 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe520009a30 con 0x7fe52c0ffe70 2023-07-14T17:40:37.587 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.600+0000 7fe528ff9700 1 --2- 172.21.15.178:0/3510455655 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe5140a1150 0x7fe5140a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:40:37.587 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.600+0000 7fe528ff9700 1 -- 172.21.15.178:0/3510455655 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5426+0+0 (secure 0 0 0) 0x7fe5200cbba0 con 0x7fe52c0ffe70 2023-07-14T17:40:37.587 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.600+0000 7fe52b7fe700 1 --2- 172.21.15.178:0/3510455655 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe5140a1150 0x7fe5140a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:40:37.587 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.600+0000 7fe52b7fe700 1 --2- 172.21.15.178:0/3510455655 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe5140a1150 0x7fe5140a3600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe51c00b5c0 tx=0x7fe51c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:40:37.588 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.604+0000 7fe528ff9700 1 -- 172.21.15.178:0/3510455655 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe520095e30 con 0x7fe52c0ffe70 2023-07-14T17:40:37.784 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:37.799+0000 7fe53218d700 1 -- 172.21.15.178:0/3510455655 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe52c069260 con 0x7fe5140a1150 2023-07-14T17:40:38.880 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:38 smithi178 ceph-mon[129368]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T17:40:38.880 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:38 smithi178 ceph-mon[129368]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:40:38.881 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:38 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:40:38.881 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:38 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:40:38.881 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:38 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:38.881 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:38 smithi178 ceph-mon[129368]: pgmap v158: 1 pgs: 1 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:38.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:38 smithi088 ceph-mon[127465]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T17:40:38.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:38 smithi088 ceph-mon[127465]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:40:38.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:38 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:40:38.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:38 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:40:38.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:38 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:38.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:38 smithi088 ceph-mon[127465]: pgmap v158: 1 pgs: 1 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:38.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:38 smithi107 ceph-mon[128999]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T17:40:38.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:38 smithi107 ceph-mon[128999]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:40:38.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:38 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:40:38.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:38 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:40:38.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:38 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:38.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:38 smithi107 ceph-mon[128999]: pgmap v158: 1 pgs: 1 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:39.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:39 smithi178 ceph-mon[129368]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T17:40:39.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:39 smithi088 ceph-mon[127465]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T17:40:39.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:39 smithi107 ceph-mon[128999]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T17:40:40.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:40 smithi088 ceph-mon[127465]: osdmap e53: 8 total, 8 up, 8 in 2023-07-14T17:40:40.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:40 smithi088 ceph-mon[127465]: pgmap v161: 1 pgs: 1 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:40.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:40 smithi088 ceph-mon[127465]: from='client.? 172.21.15.178:0/2298524016' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81ec848e-9acc-40e8-8890-3d80f6844597"}]: dispatch 2023-07-14T17:40:40.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:40 smithi088 ceph-mon[127465]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81ec848e-9acc-40e8-8890-3d80f6844597"}]: dispatch 2023-07-14T17:40:40.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:40 smithi088 ceph-mon[127465]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81ec848e-9acc-40e8-8890-3d80f6844597"}]': finished 2023-07-14T17:40:40.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:40 smithi088 ceph-mon[127465]: osdmap e54: 9 total, 8 up, 9 in 2023-07-14T17:40:40.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T17:40:40.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:40 smithi107 ceph-mon[128999]: osdmap e53: 8 total, 8 up, 8 in 2023-07-14T17:40:40.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:40 smithi107 ceph-mon[128999]: pgmap v161: 1 pgs: 1 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:40.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:40 smithi107 ceph-mon[128999]: from='client.? 172.21.15.178:0/2298524016' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81ec848e-9acc-40e8-8890-3d80f6844597"}]: dispatch 2023-07-14T17:40:40.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:40 smithi107 ceph-mon[128999]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81ec848e-9acc-40e8-8890-3d80f6844597"}]: dispatch 2023-07-14T17:40:40.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:40 smithi107 ceph-mon[128999]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81ec848e-9acc-40e8-8890-3d80f6844597"}]': finished 2023-07-14T17:40:40.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:40 smithi107 ceph-mon[128999]: osdmap e54: 9 total, 8 up, 9 in 2023-07-14T17:40:40.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T17:40:41.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:40 smithi178 ceph-mon[129368]: osdmap e53: 8 total, 8 up, 8 in 2023-07-14T17:40:41.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:40 smithi178 ceph-mon[129368]: pgmap v161: 1 pgs: 1 unknown; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:41.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:40 smithi178 ceph-mon[129368]: from='client.? 172.21.15.178:0/2298524016' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81ec848e-9acc-40e8-8890-3d80f6844597"}]: dispatch 2023-07-14T17:40:41.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:40 smithi178 ceph-mon[129368]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "81ec848e-9acc-40e8-8890-3d80f6844597"}]: dispatch 2023-07-14T17:40:41.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:40 smithi178 ceph-mon[129368]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "81ec848e-9acc-40e8-8890-3d80f6844597"}]': finished 2023-07-14T17:40:41.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:40 smithi178 ceph-mon[129368]: osdmap e54: 9 total, 8 up, 9 in 2023-07-14T17:40:41.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T17:40:41.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:41 smithi088 ceph-mon[127465]: from='client.? 172.21.15.178:0/3369129037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:40:41.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:41 smithi107 ceph-mon[128999]: from='client.? 172.21.15.178:0/3369129037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:40:42.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:41 smithi178 ceph-mon[129368]: from='client.? 172.21.15.178:0/3369129037' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:40:42.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:42 smithi088 ceph-mon[127465]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:42.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:42 smithi107 ceph-mon[128999]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:43.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:42 smithi178 ceph-mon[129368]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:45.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:44 smithi178 ceph-mon[129368]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:45.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:44 smithi088 ceph-mon[127465]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:45.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:44 smithi107 ceph-mon[128999]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:46.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:46 smithi178 ceph-mon[129368]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:46.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:46 smithi088 ceph-mon[127465]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:46.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:46 smithi107 ceph-mon[128999]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:47.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:47 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T17:40:47.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:47 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:47.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:47 smithi178 ceph-mon[129368]: Deploying daemon osd.8 on smithi178 2023-07-14T17:40:47.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:47 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T17:40:47.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:47 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:47.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:47 smithi088 ceph-mon[127465]: Deploying daemon osd.8 on smithi178 2023-07-14T17:40:47.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:47 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T17:40:47.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:47 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:47.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:47 smithi107 ceph-mon[128999]: Deploying daemon osd.8 on smithi178 2023-07-14T17:40:48.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:48 smithi178 ceph-mon[129368]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:48.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:48 smithi088 ceph-mon[127465]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:48.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:48 smithi107 ceph-mon[128999]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:51.143 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:50 smithi178 ceph-mon[129368]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:51.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:50 smithi088 ceph-mon[127465]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:51.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:50 smithi107 ceph-mon[128999]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:52.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:52 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:52.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:52 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:52.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:52 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:40:52.693 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:52 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:52.693 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:52 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:52.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:52 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:40:52.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:52 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:52.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:52 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:52.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:52 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:40:53.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:53 smithi178 ceph-mon[129368]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:53.693 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:53 smithi088 ceph-mon[127465]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:53.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:53 smithi107 ceph-mon[128999]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:54.043 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:54.059+0000 7fe528ff9700 1 -- 172.21.15.178:0/3510455655 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe52c069260 con 0x7fe5140a1150 2023-07-14T17:40:54.043 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 8 on host 'smithi178' 2023-07-14T17:40:54.045 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:54.061+0000 7fe53218d700 1 -- 172.21.15.178:0/3510455655 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe5140a1150 msgr2=0x7fe5140a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:40:54.045 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:54.061+0000 7fe53218d700 1 --2- 172.21.15.178:0/3510455655 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe5140a1150 0x7fe5140a3600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe51c00b5c0 tx=0x7fe51c005fb0 comp rx=0 tx=0).stop 2023-07-14T17:40:54.045 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:54.061+0000 7fe53218d700 1 -- 172.21.15.178:0/3510455655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe52c0ffe70 msgr2=0x7fe52c194cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:40:54.045 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:54.061+0000 7fe53218d700 1 --2- 172.21.15.178:0/3510455655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe52c0ffe70 0x7fe52c194cf0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe52000d980 tx=0x7fe52000dc90 comp rx=0 tx=0).stop 2023-07-14T17:40:54.046 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:54.061+0000 7fe53218d700 1 -- 172.21.15.178:0/3510455655 shutdown_connections 2023-07-14T17:40:54.046 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:54.061+0000 7fe53218d700 1 --2- 172.21.15.178:0/3510455655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe52c100820 0x7fe52c19bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:54.046 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:54.061+0000 7fe53218d700 1 --2- 172.21.15.178:0/3510455655 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe5140a1150 0x7fe5140a3600 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:54.046 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:54.061+0000 7fe53218d700 1 --2- 172.21.15.178:0/3510455655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe52c0febd0 0x7fe52c1947b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:54.046 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:54.061+0000 7fe53218d700 1 --2- 172.21.15.178:0/3510455655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe52c0ffe70 0x7fe52c194cf0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:54.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:54.061+0000 7fe53218d700 1 -- 172.21.15.178:0/3510455655 >> 172.21.15.178:0/3510455655 conn(0x7fe52c0fa240 msgr2=0x7fe52c1033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:40:54.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:54.061+0000 7fe53218d700 1 -- 172.21.15.178:0/3510455655 shutdown_connections 2023-07-14T17:40:54.047 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:40:54.061+0000 7fe53218d700 1 -- 172.21.15.178:0/3510455655 wait complete. 2023-07-14T17:40:54.475 DEBUG:teuthology.orchestra.run.smithi178:osd.8> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.8.service 2023-07-14T17:40:54.477 INFO:tasks.cephadm:Deploying osd.9 on smithi088 with /dev/vg_nvme/lv_1... 2023-07-14T17:40:54.477 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T17:40:54.508 INFO:journalctl@ceph.osd.8.smithi178.stdout:-- Logs begin at Fri 2023-07-14 17:14:18 UTC. -- 2023-07-14T17:40:54.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:54.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:54.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:54.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:40:54.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:54.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:54 smithi178 ceph-mon[129368]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:54.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:54.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:54.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:54 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:40:54.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:54.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:54.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:54.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:40:54.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:54.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:54 smithi088 ceph-mon[127465]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:54.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:54.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:54.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:54 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:40:54.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:54.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:54.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:54.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:40:54.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:54.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:54 smithi107 ceph-mon[128999]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:54.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:54.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:54.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:54 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:40:55.048 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:40:55.253 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:40:55 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:40:55.202+0000 7fa3acde4640 -1 osd.8 0 log_to_monitors true 2023-07-14T17:40:55.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:55 smithi178 ceph-mon[129368]: from='osd.8 [v2:172.21.15.178:6816/2305567673,v1:172.21.15.178:6817/2305567673]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T17:40:55.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:55 smithi178 ceph-mon[129368]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T17:40:55.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:55 smithi088 ceph-mon[127465]: from='osd.8 [v2:172.21.15.178:6816/2305567673,v1:172.21.15.178:6817/2305567673]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T17:40:55.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:55 smithi088 ceph-mon[127465]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T17:40:55.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:55 smithi107 ceph-mon[128999]: from='osd.8 [v2:172.21.15.178:6816/2305567673,v1:172.21.15.178:6817/2305567673]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T17:40:55.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:55 smithi107 ceph-mon[128999]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T17:40:56.574 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T17:40:56.575 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T17:40:56.575 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2023-07-14T17:40:56.575 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2023-07-14T17:40:56.575 INFO:teuthology.orchestra.run.smithi088.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0143745 s, 729 MB/s 2023-07-14T17:40:56.575 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2023-07-14T17:40:56.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:56 smithi178 ceph-mon[129368]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T17:40:56.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:56 smithi178 ceph-mon[129368]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T17:40:56.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:56 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T17:40:56.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:56 smithi178 ceph-mon[129368]: from='osd.8 [v2:172.21.15.178:6816/2305567673,v1:172.21.15.178:6817/2305567673]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:40:56.595 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:56 smithi178 ceph-mon[129368]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:40:56.596 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:56 smithi178 ceph-mon[129368]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:56.915 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:40:56 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:40:56.615+0000 7fa3a2c5d700 -1 osd.8 0 waiting for initial osdmap 2023-07-14T17:40:56.916 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:40:56 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:40:56.625+0000 7fa39c24e700 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T17:40:56.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:56 smithi088 ceph-mon[127465]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T17:40:56.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:56 smithi088 ceph-mon[127465]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T17:40:56.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:56 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T17:40:56.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:56 smithi088 ceph-mon[127465]: from='osd.8 [v2:172.21.15.178:6816/2305567673,v1:172.21.15.178:6817/2305567673]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:40:56.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:56 smithi088 ceph-mon[127465]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:40:56.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:56 smithi088 ceph-mon[127465]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:56.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:56 smithi107 ceph-mon[128999]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T17:40:56.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:56 smithi107 ceph-mon[128999]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T17:40:56.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:56 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T17:40:56.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:56 smithi107 ceph-mon[128999]: from='osd.8 [v2:172.21.15.178:6816/2305567673,v1:172.21.15.178:6817/2305567673]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:40:56.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:56 smithi107 ceph-mon[128999]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:40:56.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:56 smithi107 ceph-mon[128999]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:56.991 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi088:vg_nvme/lv_1 2023-07-14T17:40:57.543 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:40:57.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:57 smithi178 ceph-mon[129368]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2023-07-14T17:40:57.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:57 smithi178 ceph-mon[129368]: osdmap e56: 9 total, 8 up, 9 in 2023-07-14T17:40:57.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:57 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T17:40:57.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:57 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T17:40:57.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:57 smithi178 ceph-mon[129368]: Detected new or changed devices on smithi178 2023-07-14T17:40:57.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:57 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:57.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:57 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:57.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:57 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:57.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:57 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:57.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:57 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:57.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:57 smithi178 ceph-mon[129368]: Adjusting osd_memory_target on smithi178 to 5722M 2023-07-14T17:40:57.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:57 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:57.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:57 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:57.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:57 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:40:57.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:57 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:57.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:57 smithi088 ceph-mon[127465]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2023-07-14T17:40:57.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:57 smithi088 ceph-mon[127465]: osdmap e56: 9 total, 8 up, 9 in 2023-07-14T17:40:57.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:57 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T17:40:57.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:57 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T17:40:57.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:57 smithi088 ceph-mon[127465]: Detected new or changed devices on smithi178 2023-07-14T17:40:57.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:57 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:57.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:57 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:57.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:57 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:57.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:57 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:57.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:57 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:57.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:57 smithi088 ceph-mon[127465]: Adjusting osd_memory_target on smithi178 to 5722M 2023-07-14T17:40:57.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:57 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:57.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:57 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:57.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:57 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:40:57.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:57 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:57.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:57 smithi107 ceph-mon[128999]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2023-07-14T17:40:57.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:57 smithi107 ceph-mon[128999]: osdmap e56: 9 total, 8 up, 9 in 2023-07-14T17:40:57.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:57 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T17:40:57.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:57 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T17:40:57.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:57 smithi107 ceph-mon[128999]: Detected new or changed devices on smithi178 2023-07-14T17:40:57.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:57 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:57.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:57 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:57.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:57 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:57.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:57 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:57.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:57 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:40:57.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:57 smithi107 ceph-mon[128999]: Adjusting osd_memory_target on smithi178 to 5722M 2023-07-14T17:40:57.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:57 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:57.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:57 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:57.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:57 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:40:57.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:57 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:40:58.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.741+0000 7f39aa268700 1 -- 172.21.15.88:0/3577663703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a4100740 msgr2=0x7f39a4107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:40:58.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.741+0000 7f39aa268700 1 --2- 172.21.15.88:0/3577663703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a4100740 0x7f39a4107000 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f399c00b0a0 tx=0x7f399c00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:40:58.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.741+0000 7f39aa268700 1 -- 172.21.15.88:0/3577663703 shutdown_connections 2023-07-14T17:40:58.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.741+0000 7f39aa268700 1 --2- 172.21.15.88:0/3577663703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a4100740 0x7f39a4107000 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:58.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.741+0000 7f39aa268700 1 --2- 172.21.15.88:0/3577663703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f39a40febb0 0x7f39a40fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:58.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.741+0000 7f39aa268700 1 --2- 172.21.15.88:0/3577663703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39a40ffdb0 0x7f39a4100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:58.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.741+0000 7f39aa268700 1 -- 172.21.15.88:0/3577663703 >> 172.21.15.88:0/3577663703 conn(0x7f39a40fa1e0 msgr2=0x7f39a40fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:40:58.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.741+0000 7f39aa268700 1 -- 172.21.15.88:0/3577663703 shutdown_connections 2023-07-14T17:40:58.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.741+0000 7f39aa268700 1 -- 172.21.15.88:0/3577663703 wait complete. 2023-07-14T17:40:58.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.742+0000 7f39aa268700 1 Processor -- start 2023-07-14T17:40:58.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.742+0000 7f39aa268700 1 -- start start 2023-07-14T17:40:58.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.742+0000 7f39aa268700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a40febb0 0x7f39a4194840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:40:58.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.742+0000 7f39aa268700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f39a40ffdb0 0x7f39a4194d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:40:58.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.742+0000 7f39aa268700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39a4100740 0x7f39a419be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:40:58.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.742+0000 7f39aa268700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a419c3d0 con 0x7f39a40febb0 2023-07-14T17:40:58.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.742+0000 7f39aa268700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a419c510 con 0x7f39a4100740 2023-07-14T17:40:58.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.742+0000 7f39aa268700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a4068180 con 0x7f39a40ffdb0 2023-07-14T17:40:58.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.742+0000 7f39a3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a40febb0 0x7f39a4194840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:40:58.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.742+0000 7f39a3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a40febb0 0x7f39a4194840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:40892/0 (socket says 172.21.15.88:40892) 2023-07-14T17:40:58.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.742+0000 7f39a3fff700 1 -- 172.21.15.88:0/4269990728 learned_addr learned my addr 172.21.15.88:0/4269990728 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:40:58.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.742+0000 7f39a37fe700 1 --2- 172.21.15.88:0/4269990728 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f39a40ffdb0 0x7f39a4194d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:40:58.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.742+0000 7f39a3fff700 1 -- 172.21.15.88:0/4269990728 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f39a40ffdb0 msgr2=0x7f39a4194d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:40:58.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.742+0000 7f39a3fff700 1 --2- 172.21.15.88:0/4269990728 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f39a40ffdb0 0x7f39a4194d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:58.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.742+0000 7f39a3fff700 1 -- 172.21.15.88:0/4269990728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39a4100740 msgr2=0x7f39a419be00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T17:40:58.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.742+0000 7f39a3fff700 1 --2- 172.21.15.88:0/4269990728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39a4100740 0x7f39a419be00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:40:58.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.742+0000 7f39a3fff700 1 -- 172.21.15.88:0/4269990728 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399c009d00 con 0x7f39a40febb0 2023-07-14T17:40:58.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.743+0000 7f39a3fff700 1 --2- 172.21.15.88:0/4269990728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a40febb0 0x7f39a4194840 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f398c00d980 tx=0x7f398c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:40:58.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.743+0000 7f39a17fa700 1 -- 172.21.15.88:0/4269990728 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c004020 con 0x7f39a40febb0 2023-07-14T17:40:58.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.743+0000 7f39aa268700 1 -- 172.21.15.88:0/4269990728 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a4068460 con 0x7f39a40febb0 2023-07-14T17:40:58.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.743+0000 7f39a17fa700 1 -- 172.21.15.88:0/4269990728 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f398c009730 con 0x7f39a40febb0 2023-07-14T17:40:58.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.743+0000 7f39a17fa700 1 -- 172.21.15.88:0/4269990728 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f398c0039c0 con 0x7f39a40febb0 2023-07-14T17:40:58.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.743+0000 7f39aa268700 1 -- 172.21.15.88:0/4269990728 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a40689b0 con 0x7f39a40febb0 2023-07-14T17:40:58.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.744+0000 7f39aa268700 1 -- 172.21.15.88:0/4269990728 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a4106130 con 0x7f39a40febb0 2023-07-14T17:40:58.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.745+0000 7f39a17fa700 1 -- 172.21.15.88:0/4269990728 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f398c0098a0 con 0x7f39a40febb0 2023-07-14T17:40:58.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.745+0000 7f39a17fa700 1 --2- 172.21.15.88:0/4269990728 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f39900a1150 0x7f39900a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:40:58.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.746+0000 7f39a37fe700 1 --2- 172.21.15.88:0/4269990728 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f39900a1150 0x7f39900a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:40:58.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.746+0000 7f39a17fa700 1 -- 172.21.15.88:0/4269990728 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5858+0+0 (secure 0 0 0) 0x7f398c0cbc50 con 0x7f39a40febb0 2023-07-14T17:40:58.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.746+0000 7f39a37fe700 1 --2- 172.21.15.88:0/4269990728 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f39900a1150 0x7f39900a3600 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f399400ac50 tx=0x7f399400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:40:58.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.750+0000 7f39a17fa700 1 -- 172.21.15.88:0/4269990728 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f398c0d1020 con 0x7f39a40febb0 2023-07-14T17:40:58.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:58 smithi178 ceph-mon[129368]: purged_snaps scrub starts 2023-07-14T17:40:58.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:58 smithi178 ceph-mon[129368]: purged_snaps scrub ok 2023-07-14T17:40:58.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:58 smithi178 ceph-mon[129368]: osd.8 [v2:172.21.15.178:6816/2305567673,v1:172.21.15.178:6817/2305567673] boot 2023-07-14T17:40:58.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:58 smithi178 ceph-mon[129368]: osdmap e57: 9 total, 9 up, 9 in 2023-07-14T17:40:58.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:58 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T17:40:58.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:58 smithi178 ceph-mon[129368]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:58.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:40:58.946+0000 7f39aa268700 1 -- 172.21.15.88:0/4269990728 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f39a402c980 con 0x7f39900a1150 2023-07-14T17:40:58.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:58 smithi088 ceph-mon[127465]: purged_snaps scrub starts 2023-07-14T17:40:58.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:58 smithi088 ceph-mon[127465]: purged_snaps scrub ok 2023-07-14T17:40:58.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:58 smithi088 ceph-mon[127465]: osd.8 [v2:172.21.15.178:6816/2305567673,v1:172.21.15.178:6817/2305567673] boot 2023-07-14T17:40:58.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:58 smithi088 ceph-mon[127465]: osdmap e57: 9 total, 9 up, 9 in 2023-07-14T17:40:58.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:58 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T17:40:58.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:58 smithi088 ceph-mon[127465]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:58.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:58 smithi107 ceph-mon[128999]: purged_snaps scrub starts 2023-07-14T17:40:58.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:58 smithi107 ceph-mon[128999]: purged_snaps scrub ok 2023-07-14T17:40:58.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:58 smithi107 ceph-mon[128999]: osd.8 [v2:172.21.15.178:6816/2305567673,v1:172.21.15.178:6817/2305567673] boot 2023-07-14T17:40:58.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:58 smithi107 ceph-mon[128999]: osdmap e57: 9 total, 9 up, 9 in 2023-07-14T17:40:58.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:58 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T17:40:58.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:58 smithi107 ceph-mon[128999]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T17:40:59.905 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:59 smithi088 ceph-mon[127465]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T17:40:59.905 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:59 smithi088 ceph-mon[127465]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:40:59.905 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:59 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:40:59.905 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:59 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:40:59.906 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:40:59 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:59.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:59 smithi178 ceph-mon[129368]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T17:40:59.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:59 smithi178 ceph-mon[129368]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:40:59.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:59 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:40:59.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:59 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:40:59.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:40:59 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:40:59.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:59 smithi107 ceph-mon[128999]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T17:40:59.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:59 smithi107 ceph-mon[128999]: from='client.14418 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:40:59.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:59 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:40:59.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:59 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:40:59.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:40:59 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:00.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:00 smithi178 ceph-mon[129368]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T17:41:00.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:00 smithi178 ceph-mon[129368]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T17:41:00.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:00 smithi088 ceph-mon[127465]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T17:41:00.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:00 smithi088 ceph-mon[127465]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T17:41:00.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:00 smithi107 ceph-mon[128999]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T17:41:00.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:00 smithi107 ceph-mon[128999]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T17:41:01.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:01 smithi178 ceph-mon[129368]: osdmap e60: 9 total, 9 up, 9 in 2023-07-14T17:41:01.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:01 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1601857809' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96a3d388-7f40-40ba-80ac-b670a13a9993"}]: dispatch 2023-07-14T17:41:01.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:01 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1601857809' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96a3d388-7f40-40ba-80ac-b670a13a9993"}]': finished 2023-07-14T17:41:01.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:01 smithi178 ceph-mon[129368]: osdmap e61: 10 total, 9 up, 10 in 2023-07-14T17:41:01.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:01 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T17:41:01.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:01 smithi088 ceph-mon[127465]: osdmap e60: 9 total, 9 up, 9 in 2023-07-14T17:41:01.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:01 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1601857809' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96a3d388-7f40-40ba-80ac-b670a13a9993"}]: dispatch 2023-07-14T17:41:01.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:01 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1601857809' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96a3d388-7f40-40ba-80ac-b670a13a9993"}]': finished 2023-07-14T17:41:01.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:01 smithi088 ceph-mon[127465]: osdmap e61: 10 total, 9 up, 10 in 2023-07-14T17:41:01.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:01 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T17:41:01.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:01 smithi107 ceph-mon[128999]: osdmap e60: 9 total, 9 up, 9 in 2023-07-14T17:41:01.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:01 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1601857809' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "96a3d388-7f40-40ba-80ac-b670a13a9993"}]: dispatch 2023-07-14T17:41:01.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:01 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1601857809' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "96a3d388-7f40-40ba-80ac-b670a13a9993"}]': finished 2023-07-14T17:41:01.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:01 smithi107 ceph-mon[128999]: osdmap e61: 10 total, 9 up, 10 in 2023-07-14T17:41:01.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:01 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T17:41:02.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:02 smithi178 ceph-mon[129368]: pgmap v180: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T17:41:02.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:02 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1848244933' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:41:02.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:02 smithi088 ceph-mon[127465]: pgmap v180: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T17:41:02.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:02 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1848244933' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:41:02.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:02 smithi107 ceph-mon[128999]: pgmap v180: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T17:41:02.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:02 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1848244933' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:41:05.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:04 smithi178 ceph-mon[129368]: pgmap v181: 1 pgs: 1 peering; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 110 KiB/s, 0 objects/s recovering 2023-07-14T17:41:05.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:04 smithi088 ceph-mon[127465]: pgmap v181: 1 pgs: 1 peering; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 110 KiB/s, 0 objects/s recovering 2023-07-14T17:41:05.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:04 smithi107 ceph-mon[128999]: pgmap v181: 1 pgs: 1 peering; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 110 KiB/s, 0 objects/s recovering 2023-07-14T17:41:06.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:06 smithi178 ceph-mon[129368]: pgmap v182: 1 pgs: 1 peering; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-07-14T17:41:06.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:06 smithi088 ceph-mon[127465]: pgmap v182: 1 pgs: 1 peering; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-07-14T17:41:06.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:06 smithi107 ceph-mon[128999]: pgmap v182: 1 pgs: 1 peering; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 93 KiB/s, 0 objects/s recovering 2023-07-14T17:41:07.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:07 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T17:41:07.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:07 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:08.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:07 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T17:41:08.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:07 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:08.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:07 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T17:41:08.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:07 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:09.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:08 smithi178 ceph-mon[129368]: Deploying daemon osd.9 on smithi088 2023-07-14T17:41:09.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:08 smithi178 ceph-mon[129368]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T17:41:09.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:08 smithi088 ceph-mon[127465]: Deploying daemon osd.9 on smithi088 2023-07-14T17:41:09.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:08 smithi088 ceph-mon[127465]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T17:41:09.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:08 smithi107 ceph-mon[128999]: Deploying daemon osd.9 on smithi088 2023-07-14T17:41:09.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:08 smithi107 ceph-mon[128999]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T17:41:11.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:10 smithi178 ceph-mon[129368]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-14T17:41:11.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:10 smithi088 ceph-mon[127465]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-14T17:41:11.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:10 smithi107 ceph-mon[128999]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-14T17:41:13.255 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:13 smithi088 ceph-mon[127465]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T17:41:13.256 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:13 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:13.256 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:13 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:13.256 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:13 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:41:13.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:13 smithi107 ceph-mon[128999]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T17:41:13.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:13 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:13.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:13 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:13.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:13 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:41:13.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:13 smithi178 ceph-mon[129368]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T17:41:13.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:13 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:13.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:13 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:13.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:13 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:41:14.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:41:14.895+0000 7f39a17fa700 1 -- 172.21.15.88:0/4269990728 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f39a402c980 con 0x7f39900a1150 2023-07-14T17:41:14.878 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 9 on host 'smithi088' 2023-07-14T17:41:14.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:41:14.897+0000 7f39aa268700 1 -- 172.21.15.88:0/4269990728 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f39900a1150 msgr2=0x7f39900a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:41:14.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:41:14.897+0000 7f39aa268700 1 --2- 172.21.15.88:0/4269990728 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f39900a1150 0x7f39900a3600 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f399400ac50 tx=0x7f399400a380 comp rx=0 tx=0).stop 2023-07-14T17:41:14.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:41:14.897+0000 7f39aa268700 1 -- 172.21.15.88:0/4269990728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a40febb0 msgr2=0x7f39a4194840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:41:14.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:41:14.897+0000 7f39aa268700 1 --2- 172.21.15.88:0/4269990728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a40febb0 0x7f39a4194840 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f398c00d980 tx=0x7f398c00dc90 comp rx=0 tx=0).stop 2023-07-14T17:41:14.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:41:14.897+0000 7f39aa268700 1 -- 172.21.15.88:0/4269990728 shutdown_connections 2023-07-14T17:41:14.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:41:14.897+0000 7f39aa268700 1 --2- 172.21.15.88:0/4269990728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f39a4100740 0x7f39a419be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:14.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:41:14.897+0000 7f39aa268700 1 --2- 172.21.15.88:0/4269990728 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f39900a1150 0x7f39900a3600 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:14.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:41:14.897+0000 7f39aa268700 1 --2- 172.21.15.88:0/4269990728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39a40febb0 0x7f39a4194840 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:14.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:41:14.897+0000 7f39aa268700 1 --2- 172.21.15.88:0/4269990728 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f39a40ffdb0 0x7f39a4194d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:14.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:41:14.897+0000 7f39aa268700 1 -- 172.21.15.88:0/4269990728 >> 172.21.15.88:0/4269990728 conn(0x7f39a40fa1e0 msgr2=0x7f39a4103270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:41:14.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:41:14.897+0000 7f39aa268700 1 -- 172.21.15.88:0/4269990728 shutdown_connections 2023-07-14T17:41:14.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:41:14.897+0000 7f39aa268700 1 -- 172.21.15.88:0/4269990728 wait complete. 2023-07-14T17:41:15.320 DEBUG:teuthology.orchestra.run.smithi088:osd.9> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.9.service 2023-07-14T17:41:15.322 INFO:tasks.cephadm:Deploying osd.10 on smithi107 with /dev/vg_nvme/lv_1... 2023-07-14T17:41:15.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T17:41:15.352 INFO:journalctl@ceph.osd.9.smithi088.stdout:-- Logs begin at Fri 2023-07-14 17:14:35 UTC. -- 2023-07-14T17:41:15.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:15 smithi088 ceph-mon[127465]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T17:41:15.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:15 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:15.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:15 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:15.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:15 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:15.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:15 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:15.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:15 smithi107 ceph-mon[128999]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T17:41:15.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:15 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:15.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:15 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:15.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:15 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:15.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:15 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:15.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:15 smithi178 ceph-mon[129368]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T17:41:15.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:15 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:15.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:15 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:15.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:15 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:15.667 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:15 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:15.888 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T17:41:16.443 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:41:16 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:41:16.207+0000 7fb256c32640 -1 osd.9 0 log_to_monitors true 2023-07-14T17:41:16.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:16 smithi107 ceph-mon[128999]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T17:41:16.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:16 smithi107 ceph-mon[128999]: from='osd.9 [v2:172.21.15.88:6826/3211834989,v1:172.21.15.88:6827/3211834989]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T17:41:16.695 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:16 smithi107 ceph-mon[128999]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T17:41:16.810 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:16 smithi088 ceph-mon[127465]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T17:41:16.810 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:16 smithi088 ceph-mon[127465]: from='osd.9 [v2:172.21.15.88:6826/3211834989,v1:172.21.15.88:6827/3211834989]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T17:41:16.810 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:16 smithi088 ceph-mon[127465]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T17:41:16.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:16 smithi178 ceph-mon[129368]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T17:41:16.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:16 smithi178 ceph-mon[129368]: from='osd.9 [v2:172.21.15.88:6826/3211834989,v1:172.21.15.88:6827/3211834989]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T17:41:16.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:16 smithi178 ceph-mon[129368]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T17:41:17.510 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T17:41:17.510 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T17:41:17.511 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2023-07-14T17:41:17.511 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2023-07-14T17:41:17.511 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111612 s, 939 MB/s 2023-07-14T17:41:17.512 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2023-07-14T17:41:17.839 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:17 smithi107 ceph-mon[128999]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T17:41:17.839 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:17 smithi107 ceph-mon[128999]: osdmap e62: 10 total, 9 up, 10 in 2023-07-14T17:41:17.840 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:17 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T17:41:17.840 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:17 smithi107 ceph-mon[128999]: from='osd.9 [v2:172.21.15.88:6826/3211834989,v1:172.21.15.88:6827/3211834989]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T17:41:17.840 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:17 smithi107 ceph-mon[128999]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T17:41:17.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi107:vg_nvme/lv_1 2023-07-14T17:41:17.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:17 smithi178 ceph-mon[129368]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T17:41:17.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:17 smithi178 ceph-mon[129368]: osdmap e62: 10 total, 9 up, 10 in 2023-07-14T17:41:17.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:17 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T17:41:17.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:17 smithi178 ceph-mon[129368]: from='osd.9 [v2:172.21.15.88:6826/3211834989,v1:172.21.15.88:6827/3211834989]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T17:41:17.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:17 smithi178 ceph-mon[129368]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T17:41:17.943 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:41:17 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:41:17.686+0000 7fb24c2aa700 -1 osd.9 0 waiting for initial osdmap 2023-07-14T17:41:17.943 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:41:17 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:41:17.699+0000 7fb24609c700 -1 osd.9 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T17:41:17.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:17 smithi088 ceph-mon[127465]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T17:41:17.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:17 smithi088 ceph-mon[127465]: osdmap e62: 10 total, 9 up, 10 in 2023-07-14T17:41:17.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:17 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T17:41:17.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:17 smithi088 ceph-mon[127465]: from='osd.9 [v2:172.21.15.88:6826/3211834989,v1:172.21.15.88:6827/3211834989]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T17:41:17.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:17 smithi088 ceph-mon[127465]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T17:41:18.400 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T17:41:18.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T17:41:18.941 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: osdmap e63: 10 total, 9 up, 10 in 2023-07-14T17:41:18.941 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T17:41:18.941 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T17:41:18.941 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T17:41:18.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: Detected new or changed devices on smithi088 2023-07-14T17:41:18.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:18.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:18.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:18.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:18.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:18.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:18.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: Adjusting osd_memory_target on smithi088 to 4292M 2023-07-14T17:41:18.944 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:18.944 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:18.944 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:41:18.944 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:18.945 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:41:18.945 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:18.945 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:41:18.945 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:18.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T17:41:18.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: osdmap e63: 10 total, 9 up, 10 in 2023-07-14T17:41:18.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T17:41:18.946 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T17:41:18.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T17:41:18.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: Detected new or changed devices on smithi088 2023-07-14T17:41:18.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:18.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:18.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:18.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:18.947 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:18.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:18.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: Adjusting osd_memory_target on smithi088 to 4292M 2023-07-14T17:41:18.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:18.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:18.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:41:18.948 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:18.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:41:18.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:18.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:41:18.949 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:19.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T17:41:19.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: osdmap e63: 10 total, 9 up, 10 in 2023-07-14T17:41:19.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T17:41:19.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T17:41:19.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T17:41:19.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: Detected new or changed devices on smithi088 2023-07-14T17:41:19.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:19.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:19.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:19.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:19.168 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:19.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:19.169 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: Adjusting osd_memory_target on smithi088 to 4292M 2023-07-14T17:41:19.170 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:19.170 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:19.171 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:41:19.171 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:19.171 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:41:19.172 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:19.172 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:41:19.173 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:19.719 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.735+0000 7f030f557700 1 -- 172.21.15.107:0/3366693708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03080fedd0 msgr2=0x7f03080ff240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:41:19.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.735+0000 7f030f557700 1 --2- 172.21.15.107:0/3366693708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03080fedd0 0x7f03080ff240 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f03040099c0 tx=0x7f0304009cd0 comp rx=0 tx=0).stop 2023-07-14T17:41:19.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.735+0000 7f030f557700 1 -- 172.21.15.107:0/3366693708 shutdown_connections 2023-07-14T17:41:19.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.735+0000 7f030f557700 1 --2- 172.21.15.107:0/3366693708 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03080ff780 0x7f0308107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:19.720 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.735+0000 7f030f557700 1 --2- 172.21.15.107:0/3366693708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0308100170 0x7f0308100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:19.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.735+0000 7f030f557700 1 --2- 172.21.15.107:0/3366693708 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03080fedd0 0x7f03080ff240 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:19.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.735+0000 7f030f557700 1 -- 172.21.15.107:0/3366693708 >> 172.21.15.107:0/3366693708 conn(0x7f03080f9990 msgr2=0x7f03080fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:41:19.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.736+0000 7f030f557700 1 -- 172.21.15.107:0/3366693708 shutdown_connections 2023-07-14T17:41:19.721 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.736+0000 7f030f557700 1 -- 172.21.15.107:0/3366693708 wait complete. 2023-07-14T17:41:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.736+0000 7f030f557700 1 Processor -- start 2023-07-14T17:41:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.736+0000 7f030f557700 1 -- start start 2023-07-14T17:41:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.736+0000 7f030f557700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03080fedd0 0x7f0308197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:41:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.736+0000 7f030f557700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03080ff780 0x7f0308198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:41:19.723 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.736+0000 7f030f557700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0308100170 0x7f030819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:41:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.736+0000 7f030f557700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030819f6d0 con 0x7f0308100170 2023-07-14T17:41:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.736+0000 7f030f557700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030819f810 con 0x7f03080fedd0 2023-07-14T17:41:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.736+0000 7f030f557700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f030819f9a0 con 0x7f03080ff780 2023-07-14T17:41:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.736+0000 7f030d2f3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03080fedd0 0x7f0308197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:41:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.736+0000 7f030d2f3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03080fedd0 0x7f0308197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.107:3300/0 says I am v2:172.21.15.107:42770/0 (socket says 172.21.15.107:42770) 2023-07-14T17:41:19.724 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.736+0000 7f030d2f3700 1 -- 172.21.15.107:0/975216294 learned_addr learned my addr 172.21.15.107:0/975216294 (peer_addr_for_me v2:172.21.15.107:0/0) 2023-07-14T17:41:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.737+0000 7f030caf2700 1 --2- 172.21.15.107:0/975216294 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03080ff780 0x7f0308198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:41:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.737+0000 7f030daf4700 1 --2- 172.21.15.107:0/975216294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0308100170 0x7f030819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:41:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.737+0000 7f030d2f3700 1 -- 172.21.15.107:0/975216294 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03080ff780 msgr2=0x7f0308198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:41:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.737+0000 7f030d2f3700 1 --2- 172.21.15.107:0/975216294 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03080ff780 0x7f0308198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.737+0000 7f030d2f3700 1 -- 172.21.15.107:0/975216294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0308100170 msgr2=0x7f030819f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:41:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.737+0000 7f030d2f3700 1 --2- 172.21.15.107:0/975216294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0308100170 0x7f030819f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:19.725 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.737+0000 7f030d2f3700 1 -- 172.21.15.107:0/975216294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03040095f0 con 0x7f03080fedd0 2023-07-14T17:41:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.737+0000 7f030daf4700 1 --2- 172.21.15.107:0/975216294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0308100170 0x7f030819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:41:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.737+0000 7f030d2f3700 1 --2- 172.21.15.107:0/975216294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03080fedd0 0x7f0308197b40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f02f800b530 tx=0x7f02f800b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:41:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.737+0000 7f02fe7fc700 1 -- 172.21.15.107:0/975216294 <== mon.2 v2:172.21.15.107:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f8011830 con 0x7f03080fedd0 2023-07-14T17:41:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.737+0000 7f02fe7fc700 1 -- 172.21.15.107:0/975216294 <== mon.2 v2:172.21.15.107:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f02f8011e20 con 0x7f03080fedd0 2023-07-14T17:41:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.737+0000 7f030f557700 1 -- 172.21.15.107:0/975216294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f030819fc80 con 0x7f03080fedd0 2023-07-14T17:41:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.737+0000 7f02fe7fc700 1 -- 172.21.15.107:0/975216294 <== mon.2 v2:172.21.15.107:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f800f2e0 con 0x7f03080fedd0 2023-07-14T17:41:19.726 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.737+0000 7f030f557700 1 -- 172.21.15.107:0/975216294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03081a0180 con 0x7f03080fedd0 2023-07-14T17:41:19.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.738+0000 7f030f557700 1 -- 172.21.15.107:0/975216294 --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f030810a2d0 con 0x7f03080fedd0 2023-07-14T17:41:19.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.739+0000 7f02fe7fc700 1 -- 172.21.15.107:0/975216294 <== mon.2 v2:172.21.15.107:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f02f8011990 con 0x7f03080fedd0 2023-07-14T17:41:19.727 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.740+0000 7f02fe7fc700 1 --2- 172.21.15.107:0/975216294 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f02f40a11d0 0x7f02f40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:41:19.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.740+0000 7f02fe7fc700 1 -- 172.21.15.107:0/975216294 <== mon.2 v2:172.21.15.107:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f02f80cbf70 con 0x7f03080fedd0 2023-07-14T17:41:19.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.740+0000 7f030caf2700 1 --2- 172.21.15.107:0/975216294 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f02f40a11d0 0x7f02f40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:41:19.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.740+0000 7f030caf2700 1 --2- 172.21.15.107:0/975216294 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f02f40a11d0 0x7f02f40a3680 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f030400b580 tx=0x7f0304005bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:41:19.728 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.743+0000 7f02fe7fc700 1 -- 172.21.15.107:0/975216294 <== mon.2 v2:172.21.15.107:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f02f8095ec0 con 0x7f03080fedd0 2023-07-14T17:41:19.920 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:19.936+0000 7f030f557700 1 -- 172.21.15.107:0/975216294 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f03080fe410 con 0x7f02f40a11d0 2023-07-14T17:41:19.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:19 smithi088 ceph-mon[127465]: purged_snaps scrub starts 2023-07-14T17:41:19.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:19 smithi088 ceph-mon[127465]: purged_snaps scrub ok 2023-07-14T17:41:19.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:19 smithi088 ceph-mon[127465]: osd.9 [v2:172.21.15.88:6826/3211834989,v1:172.21.15.88:6827/3211834989] boot 2023-07-14T17:41:19.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:19 smithi088 ceph-mon[127465]: osdmap e64: 10 total, 10 up, 10 in 2023-07-14T17:41:19.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:19 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T17:41:19.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:19 smithi107 ceph-mon[128999]: purged_snaps scrub starts 2023-07-14T17:41:19.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:19 smithi107 ceph-mon[128999]: purged_snaps scrub ok 2023-07-14T17:41:19.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:19 smithi107 ceph-mon[128999]: osd.9 [v2:172.21.15.88:6826/3211834989,v1:172.21.15.88:6827/3211834989] boot 2023-07-14T17:41:19.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:19 smithi107 ceph-mon[128999]: osdmap e64: 10 total, 10 up, 10 in 2023-07-14T17:41:19.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:19 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T17:41:20.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:19 smithi178 ceph-mon[129368]: purged_snaps scrub starts 2023-07-14T17:41:20.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:19 smithi178 ceph-mon[129368]: purged_snaps scrub ok 2023-07-14T17:41:20.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:19 smithi178 ceph-mon[129368]: osd.9 [v2:172.21.15.88:6826/3211834989,v1:172.21.15.88:6827/3211834989] boot 2023-07-14T17:41:20.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:19 smithi178 ceph-mon[129368]: osdmap e64: 10 total, 10 up, 10 in 2023-07-14T17:41:20.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:19 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T17:41:20.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:20 smithi107 ceph-mon[128999]: osdmap e65: 10 total, 10 up, 10 in 2023-07-14T17:41:20.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:20 smithi107 ceph-mon[128999]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:20.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:20 smithi107 ceph-mon[128999]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:41:20.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:20 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:41:20.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:20 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:41:20.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:20 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:20.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:20 smithi088 ceph-mon[127465]: osdmap e65: 10 total, 10 up, 10 in 2023-07-14T17:41:20.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:20 smithi088 ceph-mon[127465]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:20.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:20 smithi088 ceph-mon[127465]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:41:20.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:20 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:41:20.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:20 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:41:20.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:20 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:21.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:20 smithi178 ceph-mon[129368]: osdmap e65: 10 total, 10 up, 10 in 2023-07-14T17:41:21.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:20 smithi178 ceph-mon[129368]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:21.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:20 smithi178 ceph-mon[129368]: from='client.24341 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:41:21.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:41:21.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:41:21.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:23.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:22 smithi178 ceph-mon[129368]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:23.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:22 smithi178 ceph-mon[129368]: from='client.? 172.21.15.107:0/604545057' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc27ef27-7791-4c70-be0b-84e60b24af36"}]: dispatch 2023-07-14T17:41:23.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:22 smithi178 ceph-mon[129368]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc27ef27-7791-4c70-be0b-84e60b24af36"}]: dispatch 2023-07-14T17:41:23.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:22 smithi178 ceph-mon[129368]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc27ef27-7791-4c70-be0b-84e60b24af36"}]': finished 2023-07-14T17:41:23.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:22 smithi178 ceph-mon[129368]: osdmap e66: 11 total, 10 up, 11 in 2023-07-14T17:41:23.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:22 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T17:41:23.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:22 smithi088 ceph-mon[127465]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:23.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:22 smithi088 ceph-mon[127465]: from='client.? 172.21.15.107:0/604545057' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc27ef27-7791-4c70-be0b-84e60b24af36"}]: dispatch 2023-07-14T17:41:23.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:22 smithi088 ceph-mon[127465]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc27ef27-7791-4c70-be0b-84e60b24af36"}]: dispatch 2023-07-14T17:41:23.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:22 smithi088 ceph-mon[127465]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc27ef27-7791-4c70-be0b-84e60b24af36"}]': finished 2023-07-14T17:41:23.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:22 smithi088 ceph-mon[127465]: osdmap e66: 11 total, 10 up, 11 in 2023-07-14T17:41:23.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:22 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T17:41:23.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:22 smithi107 ceph-mon[128999]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:23.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:22 smithi107 ceph-mon[128999]: from='client.? 172.21.15.107:0/604545057' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc27ef27-7791-4c70-be0b-84e60b24af36"}]: dispatch 2023-07-14T17:41:23.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:22 smithi107 ceph-mon[128999]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc27ef27-7791-4c70-be0b-84e60b24af36"}]: dispatch 2023-07-14T17:41:23.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:22 smithi107 ceph-mon[128999]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc27ef27-7791-4c70-be0b-84e60b24af36"}]': finished 2023-07-14T17:41:23.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:22 smithi107 ceph-mon[128999]: osdmap e66: 11 total, 10 up, 11 in 2023-07-14T17:41:23.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:22 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T17:41:24.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:23 smithi178 ceph-mon[129368]: from='client.? 172.21.15.107:0/283412962' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:41:24.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:23 smithi088 ceph-mon[127465]: from='client.? 172.21.15.107:0/283412962' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:41:24.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:23 smithi107 ceph-mon[128999]: from='client.? 172.21.15.107:0/283412962' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:41:25.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:24 smithi178 ceph-mon[129368]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:25.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:24 smithi088 ceph-mon[127465]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:25.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:24 smithi107 ceph-mon[128999]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:26.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:26 smithi178 ceph-mon[129368]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:26.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:26 smithi088 ceph-mon[127465]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:26.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:26 smithi107 ceph-mon[128999]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:29.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:28 smithi178 ceph-mon[129368]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:29.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:28 smithi088 ceph-mon[127465]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:29.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:28 smithi107 ceph-mon[128999]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:30.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:29 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T17:41:30.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:29 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:30.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:29 smithi178 ceph-mon[129368]: Deploying daemon osd.10 on smithi107 2023-07-14T17:41:30.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:29 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T17:41:30.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:29 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:30.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:29 smithi088 ceph-mon[127465]: Deploying daemon osd.10 on smithi107 2023-07-14T17:41:30.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:29 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T17:41:30.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:29 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:30.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:29 smithi107 ceph-mon[128999]: Deploying daemon osd.10 on smithi107 2023-07-14T17:41:31.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:30 smithi178 ceph-mon[129368]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:31.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:30 smithi088 ceph-mon[127465]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:31.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:30 smithi107 ceph-mon[128999]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:33.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:32 smithi178 ceph-mon[129368]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:33.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:32 smithi088 ceph-mon[127465]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:33.207 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:32 smithi107 ceph-mon[128999]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:34.628 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:34 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:34.629 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:34 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:34.629 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:34 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:41:34.629 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:34 smithi107 ceph-mon[128999]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:34.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:34.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:34.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:41:34.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:34 smithi178 ceph-mon[129368]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:34.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:34.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:34.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:41:34.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:34 smithi088 ceph-mon[127465]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:36.330 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 10 on host 'smithi107' 2023-07-14T17:41:36.330 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:36.345+0000 7f02fe7fc700 1 -- 172.21.15.107:0/975216294 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f03080fe410 con 0x7f02f40a11d0 2023-07-14T17:41:36.332 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:36.349+0000 7f030f557700 1 -- 172.21.15.107:0/975216294 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f02f40a11d0 msgr2=0x7f02f40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:41:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:36.349+0000 7f030f557700 1 --2- 172.21.15.107:0/975216294 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f02f40a11d0 0x7f02f40a3680 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f030400b580 tx=0x7f0304005bc0 comp rx=0 tx=0).stop 2023-07-14T17:41:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:36.349+0000 7f030f557700 1 -- 172.21.15.107:0/975216294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03080fedd0 msgr2=0x7f0308197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:41:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:36.349+0000 7f030f557700 1 --2- 172.21.15.107:0/975216294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03080fedd0 0x7f0308197b40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f02f800b530 tx=0x7f02f800b840 comp rx=0 tx=0).stop 2023-07-14T17:41:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:36.349+0000 7f030f557700 1 -- 172.21.15.107:0/975216294 shutdown_connections 2023-07-14T17:41:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:36.349+0000 7f030f557700 1 --2- 172.21.15.107:0/975216294 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f02f40a11d0 0x7f02f40a3680 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:36.333 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:36.349+0000 7f030f557700 1 --2- 172.21.15.107:0/975216294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0308100170 0x7f030819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:36.349+0000 7f030f557700 1 --2- 172.21.15.107:0/975216294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03080fedd0 0x7f0308197b40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:36.349+0000 7f030f557700 1 --2- 172.21.15.107:0/975216294 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03080ff780 0x7f0308198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:36.349+0000 7f030f557700 1 -- 172.21.15.107:0/975216294 >> 172.21.15.107:0/975216294 conn(0x7f03080f9990 msgr2=0x7f03080fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:41:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:36.349+0000 7f030f557700 1 -- 172.21.15.107:0/975216294 shutdown_connections 2023-07-14T17:41:36.334 INFO:teuthology.orchestra.run.smithi107.stderr:2023-07-14T17:41:36.349+0000 7f030f557700 1 -- 172.21.15.107:0/975216294 wait complete. 2023-07-14T17:41:36.794 DEBUG:teuthology.orchestra.run.smithi107:osd.10> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.10.service 2023-07-14T17:41:36.797 INFO:tasks.cephadm:Deploying osd.11 on smithi178 with /dev/vg_nvme/lv_1... 2023-07-14T17:41:36.797 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T17:41:36.830 INFO:journalctl@ceph.osd.10.smithi107.stdout:-- Logs begin at Fri 2023-07-14 17:14:00 UTC. -- 2023-07-14T17:41:37.361 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T17:41:37.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:37 smithi178 ceph-mon[129368]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:37.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:37 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:37.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:37 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:37.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:37 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:37.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:37 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:37.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:37 smithi088 ceph-mon[127465]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:37.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:37 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:37.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:37 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:37.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:37 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:37.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:37 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:37.451 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:37 smithi107 ceph-mon[128999]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:37.451 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:37 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:37.451 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:37 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:37.451 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:37 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:37.451 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:37 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:37.725 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:41:37 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:41:37.466+0000 7fa4a9ec5640 -1 osd.10 0 log_to_monitors true 2023-07-14T17:41:38.367 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:38 smithi178 ceph-mon[129368]: from='osd.10 [v2:172.21.15.107:6824/1223599278,v1:172.21.15.107:6825/1223599278]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T17:41:38.367 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:38 smithi178 ceph-mon[129368]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T17:41:38.403 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:38 smithi107 ceph-mon[128999]: from='osd.10 [v2:172.21.15.107:6824/1223599278,v1:172.21.15.107:6825/1223599278]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T17:41:38.404 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:38 smithi107 ceph-mon[128999]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T17:41:38.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:38 smithi088 ceph-mon[127465]: from='osd.10 [v2:172.21.15.107:6824/1223599278,v1:172.21.15.107:6825/1223599278]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T17:41:38.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:38 smithi088 ceph-mon[127465]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T17:41:38.867 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T17:41:38.867 INFO:teuthology.orchestra.run.smithi178.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T17:41:38.868 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 10+0 records in 2023-07-14T17:41:38.868 INFO:teuthology.orchestra.run.smithi178.stderr:10+0 records out 2023-07-14T17:41:38.868 INFO:teuthology.orchestra.run.smithi178.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00809096 s, 1.3 GB/s 2023-07-14T17:41:38.868 INFO:teuthology.orchestra.run.smithi178.stderr: stderr: 2023-07-14T17:41:38.869 INFO:teuthology.orchestra.run.smithi178.stderr:--> Zapping successful for: 2023-07-14T17:41:39.150 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:41:39 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:41:39.155+0000 7fa49f53d700 -1 osd.10 0 waiting for initial osdmap 2023-07-14T17:41:39.151 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:39 smithi107 ceph-mon[128999]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:39.151 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:39 smithi107 ceph-mon[128999]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T17:41:39.151 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:39 smithi107 ceph-mon[128999]: osdmap e67: 11 total, 10 up, 11 in 2023-07-14T17:41:39.151 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:39 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T17:41:39.151 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:39 smithi107 ceph-mon[128999]: from='osd.10 [v2:172.21.15.107:6824/1223599278,v1:172.21.15.107:6825/1223599278]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:41:39.151 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:39 smithi107 ceph-mon[128999]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:41:39.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:39 smithi178 ceph-mon[129368]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:39.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:39 smithi178 ceph-mon[129368]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T17:41:39.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:39 smithi178 ceph-mon[129368]: osdmap e67: 11 total, 10 up, 11 in 2023-07-14T17:41:39.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:39 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T17:41:39.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:39 smithi178 ceph-mon[129368]: from='osd.10 [v2:172.21.15.107:6824/1223599278,v1:172.21.15.107:6825/1223599278]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:41:39.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:39 smithi178 ceph-mon[129368]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:41:39.218 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi178:vg_nvme/lv_1 2023-07-14T17:41:39.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:39 smithi088 ceph-mon[127465]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:39.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:39 smithi088 ceph-mon[127465]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T17:41:39.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:39 smithi088 ceph-mon[127465]: osdmap e67: 11 total, 10 up, 11 in 2023-07-14T17:41:39.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:39 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T17:41:39.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:39 smithi088 ceph-mon[127465]: from='osd.10 [v2:172.21.15.107:6824/1223599278,v1:172.21.15.107:6825/1223599278]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:41:39.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:39 smithi088 ceph-mon[127465]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2023-07-14T17:41:39.475 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:41:39 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:41:39.166+0000 7fa49832d700 -1 osd.10 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T17:41:39.777 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T17:41:40.335 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2023-07-14T17:41:40.335 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: osdmap e68: 11 total, 10 up, 11 in 2023-07-14T17:41:40.336 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T17:41:40.336 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T17:41:40.336 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: Detected new or changed devices on smithi107 2023-07-14T17:41:40.336 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:40.336 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:40.336 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:40.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:40.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:40.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:40.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:40.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:40.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:41:40.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:40.337 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:41:40.338 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:40.338 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:41:40.338 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:40.338 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:40 smithi178 ceph-mon[129368]: osd.10 [v2:172.21.15.107:6824/1223599278,v1:172.21.15.107:6825/1223599278] boot 2023-07-14T17:41:40.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2023-07-14T17:41:40.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: osdmap e68: 11 total, 10 up, 11 in 2023-07-14T17:41:40.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T17:41:40.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T17:41:40.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: Detected new or changed devices on smithi107 2023-07-14T17:41:40.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:40.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:40.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:40.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:40.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:40.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:40.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:40.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:40.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:41:40.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:40.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:41:40.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:40.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:41:40.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:40.445 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:40 smithi088 ceph-mon[127465]: osd.10 [v2:172.21.15.107:6824/1223599278,v1:172.21.15.107:6825/1223599278] boot 2023-07-14T17:41:40.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2023-07-14T17:41:40.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: osdmap e68: 11 total, 10 up, 11 in 2023-07-14T17:41:40.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T17:41:40.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T17:41:40.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: Detected new or changed devices on smithi107 2023-07-14T17:41:40.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:40.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:40.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:40.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:40.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:40.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:41:40.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:40.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:40.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:41:40.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:40.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:41:40.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:40.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:41:40.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:40.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:40 smithi107 ceph-mon[128999]: osd.10 [v2:172.21.15.107:6824/1223599278,v1:172.21.15.107:6825/1223599278] boot 2023-07-14T17:41:41.089 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.104+0000 7f51eace7700 1 -- 172.21.15.178:0/546350402 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51e40ffdb0 msgr2=0x7f51e4100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:41:41.089 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.104+0000 7f51eace7700 1 --2- 172.21.15.178:0/546350402 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51e40ffdb0 0x7f51e4100200 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f51d0009990 tx=0x7f51d0009ca0 comp rx=0 tx=0).stop 2023-07-14T17:41:41.089 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.104+0000 7f51eace7700 1 -- 172.21.15.178:0/546350402 shutdown_connections 2023-07-14T17:41:41.089 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.104+0000 7f51eace7700 1 --2- 172.21.15.178:0/546350402 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51e4100740 0x7f51e4107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:41.089 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.104+0000 7f51eace7700 1 --2- 172.21.15.178:0/546350402 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51e40febb0 0x7f51e40fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:41.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.104+0000 7f51eace7700 1 --2- 172.21.15.178:0/546350402 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51e40ffdb0 0x7f51e4100200 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:41.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.104+0000 7f51eace7700 1 -- 172.21.15.178:0/546350402 >> 172.21.15.178:0/546350402 conn(0x7f51e40fa1e0 msgr2=0x7f51e40fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:41:41.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.104+0000 7f51eace7700 1 -- 172.21.15.178:0/546350402 shutdown_connections 2023-07-14T17:41:41.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.104+0000 7f51eace7700 1 -- 172.21.15.178:0/546350402 wait complete. 2023-07-14T17:41:41.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.105+0000 7f51eace7700 1 Processor -- start 2023-07-14T17:41:41.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.105+0000 7f51eace7700 1 -- start start 2023-07-14T17:41:41.090 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.105+0000 7f51eace7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51e40febb0 0x7f51e4102b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:41:41.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.105+0000 7f51eace7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51e40ffdb0 0x7f51e4101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:41:41.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.105+0000 7f51eace7700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51e4100740 0x7f51e41016d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:41:41.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.105+0000 7f51eace7700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e4101e50 con 0x7f51e40febb0 2023-07-14T17:41:41.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.105+0000 7f51eace7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e4101fc0 con 0x7f51e40ffdb0 2023-07-14T17:41:41.091 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.105+0000 7f51eace7700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e419f9d0 con 0x7f51e4100740 2023-07-14T17:41:41.092 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.105+0000 7f51e8a83700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51e40febb0 0x7f51e4102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:41:41.093 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.105+0000 7f51e9284700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51e4100740 0x7f51e41016d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:41:41.093 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.105+0000 7f51e8a83700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51e40febb0 0x7f51e4102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.178:51830/0 (socket says 172.21.15.178:51830) 2023-07-14T17:41:41.093 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.105+0000 7f51e8a83700 1 -- 172.21.15.178:0/2053255107 learned_addr learned my addr 172.21.15.178:0/2053255107 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-07-14T17:41:41.093 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.105+0000 7f51e3fff700 1 --2- 172.21.15.178:0/2053255107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51e40ffdb0 0x7f51e4101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:41:41.093 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.105+0000 7f51e9284700 1 -- 172.21.15.178:0/2053255107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51e40ffdb0 msgr2=0x7f51e4101190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:41:41.093 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.105+0000 7f51e9284700 1 --2- 172.21.15.178:0/2053255107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51e40ffdb0 0x7f51e4101190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:41.093 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.105+0000 7f51e9284700 1 -- 172.21.15.178:0/2053255107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51e40febb0 msgr2=0x7f51e4102b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:41:41.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.105+0000 7f51e9284700 1 --2- 172.21.15.178:0/2053255107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51e40febb0 0x7f51e4102b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:41.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.105+0000 7f51e9284700 1 -- 172.21.15.178:0/2053255107 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d00095f0 con 0x7f51e4100740 2023-07-14T17:41:41.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.106+0000 7f51e9284700 1 --2- 172.21.15.178:0/2053255107 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51e4100740 0x7f51e41016d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00be20 tx=0x7f51dc009ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:41:41.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.106+0000 7f51e1ffb700 1 -- 172.21.15.178:0/2053255107 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc00cc10 con 0x7f51e4100740 2023-07-14T17:41:41.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.106+0000 7f51e1ffb700 1 -- 172.21.15.178:0/2053255107 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f51dc00cd70 con 0x7f51e4100740 2023-07-14T17:41:41.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.106+0000 7f51eace7700 1 -- 172.21.15.178:0/2053255107 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e419fcb0 con 0x7f51e4100740 2023-07-14T17:41:41.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.106+0000 7f51e1ffb700 1 -- 172.21.15.178:0/2053255107 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc0125c0 con 0x7f51e4100740 2023-07-14T17:41:41.094 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.106+0000 7f51eace7700 1 -- 172.21.15.178:0/2053255107 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e41a0160 con 0x7f51e4100740 2023-07-14T17:41:41.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.107+0000 7f51eace7700 1 -- 172.21.15.178:0/2053255107 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e410a2a0 con 0x7f51e4100740 2023-07-14T17:41:41.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.108+0000 7f51e1ffb700 1 -- 172.21.15.178:0/2053255107 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f51dc014440 con 0x7f51e4100740 2023-07-14T17:41:41.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.108+0000 7f51e1ffb700 1 --2- 172.21.15.178:0/2053255107 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f51d40a11d0 0x7f51d40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:41:41.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.109+0000 7f51e1ffb700 1 -- 172.21.15.178:0/2053255107 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f51dc006e40 con 0x7f51e4100740 2023-07-14T17:41:41.095 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.109+0000 7f51e8a83700 1 --2- 172.21.15.178:0/2053255107 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f51d40a11d0 0x7f51d40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:41:41.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.109+0000 7f51e8a83700 1 --2- 172.21.15.178:0/2053255107 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f51d40a11d0 0x7f51d40a3680 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f51d800bd70 tx=0x7f51d800b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:41:41.096 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.113+0000 7f51e1ffb700 1 -- 172.21.15.178:0/2053255107 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f51dc04e780 con 0x7f51e4100740 2023-07-14T17:41:41.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:41 smithi178 ceph-mon[129368]: purged_snaps scrub starts 2023-07-14T17:41:41.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:41 smithi178 ceph-mon[129368]: purged_snaps scrub ok 2023-07-14T17:41:41.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:41 smithi178 ceph-mon[129368]: Adjusting osd_memory_target on smithi107 to 4292M 2023-07-14T17:41:41.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:41 smithi178 ceph-mon[129368]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:41.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:41 smithi178 ceph-mon[129368]: osdmap e69: 11 total, 11 up, 11 in 2023-07-14T17:41:41.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:41 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T17:41:41.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:41 smithi178 ceph-mon[129368]: osdmap e70: 11 total, 11 up, 11 in 2023-07-14T17:41:41.295 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:41.311+0000 7f51eace7700 1 -- 172.21.15.178:0/2053255107 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f51e4069160 con 0x7f51d40a11d0 2023-07-14T17:41:41.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:41 smithi088 ceph-mon[127465]: purged_snaps scrub starts 2023-07-14T17:41:41.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:41 smithi088 ceph-mon[127465]: purged_snaps scrub ok 2023-07-14T17:41:41.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:41 smithi088 ceph-mon[127465]: Adjusting osd_memory_target on smithi107 to 4292M 2023-07-14T17:41:41.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:41 smithi088 ceph-mon[127465]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:41.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:41 smithi088 ceph-mon[127465]: osdmap e69: 11 total, 11 up, 11 in 2023-07-14T17:41:41.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:41 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T17:41:41.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:41 smithi088 ceph-mon[127465]: osdmap e70: 11 total, 11 up, 11 in 2023-07-14T17:41:41.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:41 smithi107 ceph-mon[128999]: purged_snaps scrub starts 2023-07-14T17:41:41.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:41 smithi107 ceph-mon[128999]: purged_snaps scrub ok 2023-07-14T17:41:41.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:41 smithi107 ceph-mon[128999]: Adjusting osd_memory_target on smithi107 to 4292M 2023-07-14T17:41:41.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:41 smithi107 ceph-mon[128999]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T17:41:41.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:41 smithi107 ceph-mon[128999]: osdmap e69: 11 total, 11 up, 11 in 2023-07-14T17:41:41.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:41 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T17:41:41.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:41 smithi107 ceph-mon[128999]: osdmap e70: 11 total, 11 up, 11 in 2023-07-14T17:41:42.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:42 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:41:42.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:42 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:41:42.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:42 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:42.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:42 smithi178 ceph-mon[129368]: osdmap e71: 11 total, 11 up, 11 in 2023-07-14T17:41:42.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:42 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:41:42.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:42 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:41:42.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:42 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:42.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:42 smithi088 ceph-mon[127465]: osdmap e71: 11 total, 11 up, 11 in 2023-07-14T17:41:42.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:42 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T17:41:42.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:42 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T17:41:42.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:42 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:42.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:42 smithi107 ceph-mon[128999]: osdmap e71: 11 total, 11 up, 11 in 2023-07-14T17:41:43.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:43 smithi178 ceph-mon[129368]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:41:43.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:43 smithi178 ceph-mon[129368]: pgmap v210: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T17:41:43.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:43 smithi178 ceph-mon[129368]: osdmap e72: 11 total, 11 up, 11 in 2023-07-14T17:41:43.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:43 smithi088 ceph-mon[127465]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:41:43.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:43 smithi088 ceph-mon[127465]: pgmap v210: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T17:41:43.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:43 smithi088 ceph-mon[127465]: osdmap e72: 11 total, 11 up, 11 in 2023-07-14T17:41:43.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:43 smithi107 ceph-mon[128999]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi178:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:41:43.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:43 smithi107 ceph-mon[128999]: pgmap v210: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T17:41:43.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:43 smithi107 ceph-mon[128999]: osdmap e72: 11 total, 11 up, 11 in 2023-07-14T17:41:44.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:44 smithi088 ceph-mon[127465]: osdmap e73: 11 total, 11 up, 11 in 2023-07-14T17:41:44.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:44 smithi088 ceph-mon[127465]: from='client.? 172.21.15.178:0/772887578' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba0e783d-7fbe-420c-93d9-81923f17bb9f"}]: dispatch 2023-07-14T17:41:44.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:44 smithi088 ceph-mon[127465]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba0e783d-7fbe-420c-93d9-81923f17bb9f"}]: dispatch 2023-07-14T17:41:44.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:44 smithi088 ceph-mon[127465]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba0e783d-7fbe-420c-93d9-81923f17bb9f"}]': finished 2023-07-14T17:41:44.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:44 smithi088 ceph-mon[127465]: osdmap e74: 12 total, 11 up, 12 in 2023-07-14T17:41:44.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T17:41:44.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:44 smithi088 ceph-mon[127465]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T17:41:44.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:44 smithi088 ceph-mon[127465]: from='client.? 172.21.15.178:0/2730263972' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:41:44.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:44 smithi107 ceph-mon[128999]: osdmap e73: 11 total, 11 up, 11 in 2023-07-14T17:41:44.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:44 smithi107 ceph-mon[128999]: from='client.? 172.21.15.178:0/772887578' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba0e783d-7fbe-420c-93d9-81923f17bb9f"}]: dispatch 2023-07-14T17:41:44.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:44 smithi107 ceph-mon[128999]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba0e783d-7fbe-420c-93d9-81923f17bb9f"}]: dispatch 2023-07-14T17:41:44.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:44 smithi107 ceph-mon[128999]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba0e783d-7fbe-420c-93d9-81923f17bb9f"}]': finished 2023-07-14T17:41:44.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:44 smithi107 ceph-mon[128999]: osdmap e74: 12 total, 11 up, 12 in 2023-07-14T17:41:44.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T17:41:44.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:44 smithi107 ceph-mon[128999]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T17:41:44.976 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:44 smithi107 ceph-mon[128999]: from='client.? 172.21.15.178:0/2730263972' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:41:45.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:44 smithi178 ceph-mon[129368]: osdmap e73: 11 total, 11 up, 11 in 2023-07-14T17:41:45.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:44 smithi178 ceph-mon[129368]: from='client.? 172.21.15.178:0/772887578' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba0e783d-7fbe-420c-93d9-81923f17bb9f"}]: dispatch 2023-07-14T17:41:45.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:44 smithi178 ceph-mon[129368]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba0e783d-7fbe-420c-93d9-81923f17bb9f"}]: dispatch 2023-07-14T17:41:45.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:44 smithi178 ceph-mon[129368]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba0e783d-7fbe-420c-93d9-81923f17bb9f"}]': finished 2023-07-14T17:41:45.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:44 smithi178 ceph-mon[129368]: osdmap e74: 12 total, 11 up, 12 in 2023-07-14T17:41:45.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T17:41:45.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:44 smithi178 ceph-mon[129368]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T17:41:45.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:44 smithi178 ceph-mon[129368]: from='client.? 172.21.15.178:0/2730263972' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T17:41:46.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:46 smithi088 ceph-mon[127465]: pgmap v215: 1 pgs: 1 remapped+peering; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2023-07-14T17:41:46.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:46 smithi107 ceph-mon[128999]: pgmap v215: 1 pgs: 1 remapped+peering; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2023-07-14T17:41:47.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:46 smithi178 ceph-mon[129368]: pgmap v215: 1 pgs: 1 remapped+peering; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail 2023-07-14T17:41:49.058 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:48 smithi178 ceph-mon[129368]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T17:41:49.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:48 smithi088 ceph-mon[127465]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T17:41:49.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:48 smithi107 ceph-mon[128999]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 301 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T17:41:51.156 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:50 smithi178 ceph-mon[129368]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2023-07-14T17:41:51.156 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:50 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T17:41:51.156 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:50 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:51.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:50 smithi088 ceph-mon[127465]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2023-07-14T17:41:51.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:50 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T17:41:51.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:50 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:51.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:50 smithi107 ceph-mon[128999]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2023-07-14T17:41:51.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:50 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T17:41:51.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:50 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:52.074 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:51 smithi178 ceph-mon[129368]: Deploying daemon osd.11 on smithi178 2023-07-14T17:41:52.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:51 smithi088 ceph-mon[127465]: Deploying daemon osd.11 on smithi178 2023-07-14T17:41:52.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:51 smithi107 ceph-mon[128999]: Deploying daemon osd.11 on smithi178 2023-07-14T17:41:53.070 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:52 smithi178 ceph-mon[129368]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-07-14T17:41:53.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:52 smithi088 ceph-mon[127465]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-07-14T17:41:53.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:52 smithi107 ceph-mon[128999]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-07-14T17:41:55.132 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:54 smithi178 ceph-mon[129368]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-07-14T17:41:55.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:54 smithi088 ceph-mon[127465]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-07-14T17:41:55.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:54 smithi107 ceph-mon[128999]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-07-14T17:41:56.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:56 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:56.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:56 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:56.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:56 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:41:56.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:56 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:56.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:56 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:56.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:56 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:41:56.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:56 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:56.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:56 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:56.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:56 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:41:57.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:57 smithi178 ceph-mon[129368]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T17:41:57.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:57 smithi088 ceph-mon[127465]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T17:41:57.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:57 smithi107 ceph-mon[128999]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T17:41:58.104 INFO:teuthology.orchestra.run.smithi178.stdout:Created osd(s) 11 on host 'smithi178' 2023-07-14T17:41:58.105 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:58.121+0000 7f51e1ffb700 1 -- 172.21.15.178:0/2053255107 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f51e4069160 con 0x7f51d40a11d0 2023-07-14T17:41:58.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:58.124+0000 7f51eace7700 1 -- 172.21.15.178:0/2053255107 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f51d40a11d0 msgr2=0x7f51d40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:41:58.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:58.124+0000 7f51eace7700 1 --2- 172.21.15.178:0/2053255107 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f51d40a11d0 0x7f51d40a3680 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f51d800bd70 tx=0x7f51d800b4a0 comp rx=0 tx=0).stop 2023-07-14T17:41:58.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:58.124+0000 7f51eace7700 1 -- 172.21.15.178:0/2053255107 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51e4100740 msgr2=0x7f51e41016d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:41:58.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:58.124+0000 7f51eace7700 1 --2- 172.21.15.178:0/2053255107 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51e4100740 0x7f51e41016d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00be20 tx=0x7f51dc009ed0 comp rx=0 tx=0).stop 2023-07-14T17:41:58.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:58.124+0000 7f51eace7700 1 -- 172.21.15.178:0/2053255107 shutdown_connections 2023-07-14T17:41:58.107 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:58.124+0000 7f51eace7700 1 --2- 172.21.15.178:0/2053255107 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51e4100740 0x7f51e41016d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:58.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:58.124+0000 7f51eace7700 1 --2- 172.21.15.178:0/2053255107 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f51d40a11d0 0x7f51d40a3680 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:58.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:58.124+0000 7f51eace7700 1 --2- 172.21.15.178:0/2053255107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51e40febb0 0x7f51e4102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:58.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:58.124+0000 7f51eace7700 1 --2- 172.21.15.178:0/2053255107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51e40ffdb0 0x7f51e4101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:41:58.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:58.124+0000 7f51eace7700 1 -- 172.21.15.178:0/2053255107 >> 172.21.15.178:0/2053255107 conn(0x7f51e40fa1e0 msgr2=0x7f51e4068a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:41:58.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:58.124+0000 7f51eace7700 1 -- 172.21.15.178:0/2053255107 shutdown_connections 2023-07-14T17:41:58.108 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:41:58.124+0000 7f51eace7700 1 -- 172.21.15.178:0/2053255107 wait complete. 2023-07-14T17:41:58.577 DEBUG:teuthology.orchestra.run.smithi178:osd.11> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.11.service 2023-07-14T17:41:58.580 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-07-14T17:41:58.580 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd stat -f json 2023-07-14T17:41:58.610 INFO:journalctl@ceph.osd.11.smithi178.stdout:-- Logs begin at Fri 2023-07-14 17:14:18 UTC. -- 2023-07-14T17:41:58.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:58 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:58.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:58 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:58.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:58 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:58.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:58 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:41:58.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:58 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:58.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:58 smithi178 ceph-mon[129368]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T17:41:58.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:58 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:58.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:58 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:58.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:58 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:41:58.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:58 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:58.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:58 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:58.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:58 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:58.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:58 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:41:58.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:58 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:58.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:58 smithi088 ceph-mon[127465]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T17:41:58.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:58 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:58.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:58 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:58.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:58 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:41:59.148 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:41:59.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:58 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:59.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:58 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:59.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:58 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:41:59.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:58 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:41:59.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:58 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:59.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:58 smithi107 ceph-mon[128999]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T17:41:59.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:58 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:59.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:58 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:41:59.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:58 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:41:59.356 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:41:59 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:41:59.147+0000 7ffa6dadd640 -1 osd.11 0 log_to_monitors true 2023-07-14T17:41:59.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:59 smithi088 ceph-mon[127465]: from='osd.11 [v2:172.21.15.178:6824/1862417235,v1:172.21.15.178:6825/1862417235]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T17:41:59.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:41:59 smithi088 ceph-mon[127465]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T17:42:00.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:59 smithi178 ceph-mon[129368]: from='osd.11 [v2:172.21.15.178:6824/1862417235,v1:172.21.15.178:6825/1862417235]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T17:42:00.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:41:59 smithi178 ceph-mon[129368]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T17:42:00.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:59 smithi107 ceph-mon[128999]: from='osd.11 [v2:172.21.15.178:6824/1862417235,v1:172.21.15.178:6825/1862417235]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T17:42:00.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:41:59 smithi107 ceph-mon[128999]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T17:42:00.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.324+0000 7fe53a149700 1 -- 172.21.15.88:0/4065617577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5340fee30 msgr2=0x7fe5340ff2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:00.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.324+0000 7fe53a149700 1 --2- 172.21.15.88:0/4065617577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5340fee30 0x7fe5340ff2a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe528009990 tx=0x7fe528009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:00.310 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.324+0000 7fe53a149700 1 -- 172.21.15.88:0/4065617577 shutdown_connections 2023-07-14T17:42:00.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.324+0000 7fe53a149700 1 --2- 172.21.15.88:0/4065617577 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5340ff7e0 0x7fe534107090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:00.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.324+0000 7fe53a149700 1 --2- 172.21.15.88:0/4065617577 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5341001d0 0x7fe5341005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:00.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.324+0000 7fe53a149700 1 --2- 172.21.15.88:0/4065617577 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5340fee30 0x7fe5340ff2a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:00.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.324+0000 7fe53a149700 1 -- 172.21.15.88:0/4065617577 >> 172.21.15.88:0/4065617577 conn(0x7fe5340f9990 msgr2=0x7fe5340fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:00.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.324+0000 7fe53a149700 1 -- 172.21.15.88:0/4065617577 shutdown_connections 2023-07-14T17:42:00.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.324+0000 7fe53a149700 1 -- 172.21.15.88:0/4065617577 wait complete. 2023-07-14T17:42:00.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.325+0000 7fe53a149700 1 Processor -- start 2023-07-14T17:42:00.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.325+0000 7fe53a149700 1 -- start start 2023-07-14T17:42:00.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.325+0000 7fe53a149700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5340fee30 0x7fe5341947f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:00.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.325+0000 7fe53a149700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5340ff7e0 0x7fe534194d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:00.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.325+0000 7fe53a149700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5341001d0 0x7fe53419bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:00.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.325+0000 7fe53a149700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53419c380 con 0x7fe5340ff7e0 2023-07-14T17:42:00.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.325+0000 7fe53a149700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53419c4c0 con 0x7fe5341001d0 2023-07-14T17:42:00.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.325+0000 7fe53a149700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53419c650 con 0x7fe5340fee30 2023-07-14T17:42:00.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.326+0000 7fe532ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5340ff7e0 0x7fe534194d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:00.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.326+0000 7fe5337fe700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5340fee30 0x7fe5341947f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:00.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.326+0000 7fe532ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5340ff7e0 0x7fe534194d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41040/0 (socket says 172.21.15.88:41040) 2023-07-14T17:42:00.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.326+0000 7fe532ffd700 1 -- 172.21.15.88:0/1170324487 learned_addr learned my addr 172.21.15.88:0/1170324487 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:00.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.326+0000 7fe533fff700 1 --2- 172.21.15.88:0/1170324487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5341001d0 0x7fe53419bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:00.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.326+0000 7fe532ffd700 1 -- 172.21.15.88:0/1170324487 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5340fee30 msgr2=0x7fe5341947f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:00.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.326+0000 7fe532ffd700 1 --2- 172.21.15.88:0/1170324487 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5340fee30 0x7fe5341947f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:00.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.326+0000 7fe532ffd700 1 -- 172.21.15.88:0/1170324487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5341001d0 msgr2=0x7fe53419bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:00.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.326+0000 7fe532ffd700 1 --2- 172.21.15.88:0/1170324487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5341001d0 0x7fe53419bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:00.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.326+0000 7fe532ffd700 1 -- 172.21.15.88:0/1170324487 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5280095f0 con 0x7fe5340ff7e0 2023-07-14T17:42:00.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.326+0000 7fe533fff700 1 --2- 172.21.15.88:0/1170324487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5341001d0 0x7fe53419bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:42:00.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.326+0000 7fe532ffd700 1 --2- 172.21.15.88:0/1170324487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5340ff7e0 0x7fe534194d30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe5280059d0 tx=0x7fe528003a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:00.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.326+0000 7fe530ff9700 1 -- 172.21.15.88:0/1170324487 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52801d070 con 0x7fe5340ff7e0 2023-07-14T17:42:00.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.326+0000 7fe530ff9700 1 -- 172.21.15.88:0/1170324487 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe52800bd70 con 0x7fe5340ff7e0 2023-07-14T17:42:00.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.326+0000 7fe53a149700 1 -- 172.21.15.88:0/1170324487 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5340fdd20 con 0x7fe5340ff7e0 2023-07-14T17:42:00.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.326+0000 7fe530ff9700 1 -- 172.21.15.88:0/1170324487 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe528022c60 con 0x7fe5340ff7e0 2023-07-14T17:42:00.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.326+0000 7fe53a149700 1 -- 172.21.15.88:0/1170324487 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5340fe1c0 con 0x7fe5340ff7e0 2023-07-14T17:42:00.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.327+0000 7fe53a149700 1 -- 172.21.15.88:0/1170324487 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53410a240 con 0x7fe5340ff7e0 2023-07-14T17:42:00.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.328+0000 7fe530ff9700 1 -- 172.21.15.88:0/1170324487 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe528022470 con 0x7fe5340ff7e0 2023-07-14T17:42:00.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.329+0000 7fe530ff9700 1 --2- 172.21.15.88:0/1170324487 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe51c0a3360 0x7fe51c0a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:00.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.329+0000 7fe5337fe700 1 --2- 172.21.15.88:0/1170324487 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe51c0a3360 0x7fe51c0a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:00.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.329+0000 7fe530ff9700 1 -- 172.21.15.88:0/1170324487 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6831+0+0 (secure 0 0 0) 0x7fe5280cda90 con 0x7fe5340ff7e0 2023-07-14T17:42:00.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.329+0000 7fe5337fe700 1 --2- 172.21.15.88:0/1170324487 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe51c0a3360 0x7fe51c0a5810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe534067ee0 tx=0x7fe52400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:00.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.333+0000 7fe530ff9700 1 -- 172.21.15.88:0/1170324487 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe5280977a0 con 0x7fe5340ff7e0 2023-07-14T17:42:00.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.520+0000 7fe53a149700 1 -- 172.21.15.88:0/1170324487 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fe53410a430 con 0x7fe5340ff7e0 2023-07-14T17:42:00.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.520+0000 7fe530ff9700 1 -- 172.21.15.88:0/1170324487 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v75) v1 ==== 74+0+133 (secure 0 0 0) 0x7fe52809adc0 con 0x7fe5340ff7e0 2023-07-14T17:42:00.504 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:00.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.522+0000 7fe53a149700 1 -- 172.21.15.88:0/1170324487 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe51c0a3360 msgr2=0x7fe51c0a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:00.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.522+0000 7fe53a149700 1 --2- 172.21.15.88:0/1170324487 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe51c0a3360 0x7fe51c0a5810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe534067ee0 tx=0x7fe52400a380 comp rx=0 tx=0).stop 2023-07-14T17:42:00.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.522+0000 7fe53a149700 1 -- 172.21.15.88:0/1170324487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5340ff7e0 msgr2=0x7fe534194d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:00.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.522+0000 7fe53a149700 1 --2- 172.21.15.88:0/1170324487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5340ff7e0 0x7fe534194d30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe5280059d0 tx=0x7fe528003a50 comp rx=0 tx=0).stop 2023-07-14T17:42:00.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.522+0000 7fe53a149700 1 -- 172.21.15.88:0/1170324487 shutdown_connections 2023-07-14T17:42:00.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.522+0000 7fe53a149700 1 --2- 172.21.15.88:0/1170324487 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5341001d0 0x7fe53419bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:00.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.522+0000 7fe53a149700 1 --2- 172.21.15.88:0/1170324487 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5340fee30 0x7fe5341947f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:00.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.522+0000 7fe53a149700 1 --2- 172.21.15.88:0/1170324487 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe51c0a3360 0x7fe51c0a5810 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:00.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.522+0000 7fe53a149700 1 --2- 172.21.15.88:0/1170324487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5340ff7e0 0x7fe534194d30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:00.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.522+0000 7fe53a149700 1 -- 172.21.15.88:0/1170324487 >> 172.21.15.88:0/1170324487 conn(0x7fe5340f9990 msgr2=0x7fe5340fbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:00.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.522+0000 7fe53a149700 1 -- 172.21.15.88:0/1170324487 shutdown_connections 2023-07-14T17:42:00.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:00.522+0000 7fe53a149700 1 -- 172.21.15.88:0/1170324487 wait complete. 2023-07-14T17:42:00.891 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":75,"num_osds":12,"num_up_osds":11,"osd_up_since":1689356500,"num_in_osds":12,"osd_in_since":1689356503,"num_remapped_pgs":0} 2023-07-14T17:42:01.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:42:00 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:42:00.817+0000 7ffa63155700 -1 osd.11 0 waiting for initial osdmap 2023-07-14T17:42:01.165 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:42:00 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:42:00.829+0000 7ffa5c746700 -1 osd.11 76 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T17:42:01.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:00 smithi178 ceph-mon[129368]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T17:42:01.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:00 smithi178 ceph-mon[129368]: osdmap e75: 12 total, 11 up, 12 in 2023-07-14T17:42:01.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:00 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T17:42:01.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:00 smithi178 ceph-mon[129368]: from='osd.11 [v2:172.21.15.178:6824/1862417235,v1:172.21.15.178:6825/1862417235]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:42:01.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:00 smithi178 ceph-mon[129368]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:42:01.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:00 smithi178 ceph-mon[129368]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T17:42:01.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:00 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1170324487' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T17:42:01.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:00 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:01.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:00 smithi088 ceph-mon[127465]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T17:42:01.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:00 smithi088 ceph-mon[127465]: osdmap e75: 12 total, 11 up, 12 in 2023-07-14T17:42:01.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:00 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T17:42:01.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:00 smithi088 ceph-mon[127465]: from='osd.11 [v2:172.21.15.178:6824/1862417235,v1:172.21.15.178:6825/1862417235]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:42:01.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:00 smithi088 ceph-mon[127465]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:42:01.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:00 smithi088 ceph-mon[127465]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T17:42:01.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:00 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1170324487' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T17:42:01.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:00 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:01.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:00 smithi107 ceph-mon[128999]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T17:42:01.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:00 smithi107 ceph-mon[128999]: osdmap e75: 12 total, 11 up, 12 in 2023-07-14T17:42:01.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:00 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T17:42:01.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:00 smithi107 ceph-mon[128999]: from='osd.11 [v2:172.21.15.178:6824/1862417235,v1:172.21.15.178:6825/1862417235]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:42:01.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:00 smithi107 ceph-mon[128999]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]: dispatch 2023-07-14T17:42:01.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:00 smithi107 ceph-mon[128999]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T17:42:01.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:00 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1170324487' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T17:42:01.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:00 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:01.892 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd stat -f json 2023-07-14T17:42:02.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:01 smithi178 ceph-mon[129368]: Detected new or changed devices on smithi178 2023-07-14T17:42:02.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:01 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:02.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:01 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:42:02.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:01 smithi178 ceph-mon[129368]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2023-07-14T17:42:02.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:01 smithi178 ceph-mon[129368]: osdmap e76: 12 total, 11 up, 12 in 2023-07-14T17:42:02.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:01 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:42:02.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:01 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T17:42:02.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:01 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:42:02.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:01 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T17:42:02.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:01 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:42:02.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:01 smithi178 ceph-mon[129368]: Adjusting osd_memory_target on smithi178 to 4292M 2023-07-14T17:42:02.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:01 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:02.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:01 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:02.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:01 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:42:02.167 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:01 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:02.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:01 smithi088 ceph-mon[127465]: Detected new or changed devices on smithi178 2023-07-14T17:42:02.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:01 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:02.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:01 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:42:02.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:01 smithi088 ceph-mon[127465]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2023-07-14T17:42:02.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:01 smithi088 ceph-mon[127465]: osdmap e76: 12 total, 11 up, 12 in 2023-07-14T17:42:02.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:01 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:42:02.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:01 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T17:42:02.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:01 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:42:02.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:01 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T17:42:02.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:01 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:42:02.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:01 smithi088 ceph-mon[127465]: Adjusting osd_memory_target on smithi178 to 4292M 2023-07-14T17:42:02.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:01 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:02.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:01 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:02.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:01 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:42:02.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:01 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:02.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:01 smithi107 ceph-mon[128999]: Detected new or changed devices on smithi178 2023-07-14T17:42:02.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:01 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:02.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:01 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:42:02.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:01 smithi107 ceph-mon[128999]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi178", "root=default"]}]': finished 2023-07-14T17:42:02.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:01 smithi107 ceph-mon[128999]: osdmap e76: 12 total, 11 up, 12 in 2023-07-14T17:42:02.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:01 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:42:02.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:01 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T17:42:02.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:01 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:42:02.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:01 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T17:42:02.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:01 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T17:42:02.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:01 smithi107 ceph-mon[128999]: Adjusting osd_memory_target on smithi178 to 4292M 2023-07-14T17:42:02.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:01 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:02.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:01 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:02.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:01 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:42:02.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:01 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:02.446 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:42:02.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:02 smithi088 ceph-mon[127465]: purged_snaps scrub starts 2023-07-14T17:42:02.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:02 smithi088 ceph-mon[127465]: purged_snaps scrub ok 2023-07-14T17:42:02.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:02 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T17:42:02.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:02 smithi088 ceph-mon[127465]: osd.11 [v2:172.21.15.178:6824/1862417235,v1:172.21.15.178:6825/1862417235] boot 2023-07-14T17:42:02.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:02 smithi088 ceph-mon[127465]: osdmap e77: 12 total, 12 up, 12 in 2023-07-14T17:42:02.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:02 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T17:42:02.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:02 smithi088 ceph-mon[127465]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T17:42:03.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:02 smithi178 ceph-mon[129368]: purged_snaps scrub starts 2023-07-14T17:42:03.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:02 smithi178 ceph-mon[129368]: purged_snaps scrub ok 2023-07-14T17:42:03.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:02 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T17:42:03.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:02 smithi178 ceph-mon[129368]: osd.11 [v2:172.21.15.178:6824/1862417235,v1:172.21.15.178:6825/1862417235] boot 2023-07-14T17:42:03.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:02 smithi178 ceph-mon[129368]: osdmap e77: 12 total, 12 up, 12 in 2023-07-14T17:42:03.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:02 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T17:42:03.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:02 smithi178 ceph-mon[129368]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T17:42:03.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:02 smithi107 ceph-mon[128999]: purged_snaps scrub starts 2023-07-14T17:42:03.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:02 smithi107 ceph-mon[128999]: purged_snaps scrub ok 2023-07-14T17:42:03.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:02 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T17:42:03.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:02 smithi107 ceph-mon[128999]: osd.11 [v2:172.21.15.178:6824/1862417235,v1:172.21.15.178:6825/1862417235] boot 2023-07-14T17:42:03.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:02 smithi107 ceph-mon[128999]: osdmap e77: 12 total, 12 up, 12 in 2023-07-14T17:42:03.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:02 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T17:42:03.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:02 smithi107 ceph-mon[128999]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T17:42:03.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.662+0000 7f0f33d15700 1 -- 172.21.15.88:0/953298642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f2c101850 msgr2=0x7f0f2c108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:03.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.662+0000 7f0f33d15700 1 --2- 172.21.15.88:0/953298642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f2c101850 0x7f0f2c108110 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0f2400b0a0 tx=0x7f0f2400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:03.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.662+0000 7f0f33d15700 1 -- 172.21.15.88:0/953298642 shutdown_connections 2023-07-14T17:42:03.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.662+0000 7f0f33d15700 1 --2- 172.21.15.88:0/953298642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f2c101850 0x7f0f2c108110 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:03.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.662+0000 7f0f33d15700 1 --2- 172.21.15.88:0/953298642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f2c0ffcc0 0x7f0f2c1000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:03.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.662+0000 7f0f33d15700 1 --2- 172.21.15.88:0/953298642 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f2c100ec0 0x7f0f2c101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:03.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.662+0000 7f0f33d15700 1 -- 172.21.15.88:0/953298642 >> 172.21.15.88:0/953298642 conn(0x7f0f2c0fb2d0 msgr2=0x7f0f2c0fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:03.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.663+0000 7f0f33d15700 1 -- 172.21.15.88:0/953298642 shutdown_connections 2023-07-14T17:42:03.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.663+0000 7f0f33d15700 1 -- 172.21.15.88:0/953298642 wait complete. 2023-07-14T17:42:03.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.663+0000 7f0f33d15700 1 Processor -- start 2023-07-14T17:42:03.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.663+0000 7f0f33d15700 1 -- start start 2023-07-14T17:42:03.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.663+0000 7f0f33d15700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f2c0ffcc0 0x7f0f2c195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:03.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.663+0000 7f0f33d15700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f2c100ec0 0x7f0f2c195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:03.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f33d15700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f2c101850 0x7f0f2c19d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:03.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f33d15700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c19d620 con 0x7f0f2c0ffcc0 2023-07-14T17:42:03.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f33d15700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c19d760 con 0x7f0f2c100ec0 2023-07-14T17:42:03.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f33d15700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c19d8f0 con 0x7f0f2c101850 2023-07-14T17:42:03.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f31ab1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f2c0ffcc0 0x7f0f2c195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:03.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f31ab1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f2c0ffcc0 0x7f0f2c195a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41056/0 (socket says 172.21.15.88:41056) 2023-07-14T17:42:03.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f31ab1700 1 -- 172.21.15.88:0/2057339003 learned_addr learned my addr 172.21.15.88:0/2057339003 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:03.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f312b0700 1 --2- 172.21.15.88:0/2057339003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f2c100ec0 0x7f0f2c195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:03.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f322b2700 1 --2- 172.21.15.88:0/2057339003 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f2c101850 0x7f0f2c19d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:03.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f31ab1700 1 -- 172.21.15.88:0/2057339003 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f2c101850 msgr2=0x7f0f2c19d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:03.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f31ab1700 1 --2- 172.21.15.88:0/2057339003 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f2c101850 0x7f0f2c19d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:03.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f31ab1700 1 -- 172.21.15.88:0/2057339003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f2c100ec0 msgr2=0x7f0f2c195fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:03.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f31ab1700 1 --2- 172.21.15.88:0/2057339003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f2c100ec0 0x7f0f2c195fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:03.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f31ab1700 1 -- 172.21.15.88:0/2057339003 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f24009d00 con 0x7f0f2c0ffcc0 2023-07-14T17:42:03.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f31ab1700 1 --2- 172.21.15.88:0/2057339003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f2c0ffcc0 0x7f0f2c195a90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800b530 tx=0x7f0f2800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:03.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f22ffd700 1 -- 172.21.15.88:0/2057339003 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f28011830 con 0x7f0f2c0ffcc0 2023-07-14T17:42:03.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f22ffd700 1 -- 172.21.15.88:0/2057339003 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0f28011e20 con 0x7f0f2c0ffcc0 2023-07-14T17:42:03.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f33d15700 1 -- 172.21.15.88:0/2057339003 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2c0728d0 con 0x7f0f2c0ffcc0 2023-07-14T17:42:03.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f22ffd700 1 -- 172.21.15.88:0/2057339003 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f28010dd0 con 0x7f0f2c0ffcc0 2023-07-14T17:42:03.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.664+0000 7f0f33d15700 1 -- 172.21.15.88:0/2057339003 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2c072e50 con 0x7f0f2c0ffcc0 2023-07-14T17:42:03.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.665+0000 7f0f33d15700 1 -- 172.21.15.88:0/2057339003 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f2c1072e0 con 0x7f0f2c0ffcc0 2023-07-14T17:42:03.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.666+0000 7f0f22ffd700 1 -- 172.21.15.88:0/2057339003 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0f28011990 con 0x7f0f2c0ffcc0 2023-07-14T17:42:03.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.667+0000 7f0f22ffd700 1 --2- 172.21.15.88:0/2057339003 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0f180a1150 0x7f0f180a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:03.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.667+0000 7f0f312b0700 1 --2- 172.21.15.88:0/2057339003 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0f180a1150 0x7f0f180a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:03.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.667+0000 7f0f22ffd700 1 -- 172.21.15.88:0/2057339003 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0f280cbf50 con 0x7f0f2c0ffcc0 2023-07-14T17:42:03.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.667+0000 7f0f312b0700 1 --2- 172.21.15.88:0/2057339003 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0f180a1150 0x7f0f180a3600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c005950 tx=0x7f0f1c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:03.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.671+0000 7f0f22ffd700 1 -- 172.21.15.88:0/2057339003 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0f28095b40 con 0x7f0f2c0ffcc0 2023-07-14T17:42:03.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.858+0000 7f0f33d15700 1 -- 172.21.15.88:0/2057339003 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f0f2c02ca90 con 0x7f0f2c0ffcc0 2023-07-14T17:42:03.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.859+0000 7f0f22ffd700 1 -- 172.21.15.88:0/2057339003 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v78) v1 ==== 74+0+133 (secure 0 0 0) 0x7f0f28099160 con 0x7f0f2c0ffcc0 2023-07-14T17:42:03.843 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:03.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.861+0000 7f0f33d15700 1 -- 172.21.15.88:0/2057339003 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0f180a1150 msgr2=0x7f0f180a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:03.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.861+0000 7f0f33d15700 1 --2- 172.21.15.88:0/2057339003 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0f180a1150 0x7f0f180a3600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c005950 tx=0x7f0f1c0058e0 comp rx=0 tx=0).stop 2023-07-14T17:42:03.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.861+0000 7f0f33d15700 1 -- 172.21.15.88:0/2057339003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f2c0ffcc0 msgr2=0x7f0f2c195a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:03.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.861+0000 7f0f33d15700 1 --2- 172.21.15.88:0/2057339003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f2c0ffcc0 0x7f0f2c195a90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800b530 tx=0x7f0f2800b8f0 comp rx=0 tx=0).stop 2023-07-14T17:42:03.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.861+0000 7f0f33d15700 1 -- 172.21.15.88:0/2057339003 shutdown_connections 2023-07-14T17:42:03.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.861+0000 7f0f33d15700 1 --2- 172.21.15.88:0/2057339003 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f2c101850 0x7f0f2c19d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:03.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.861+0000 7f0f33d15700 1 --2- 172.21.15.88:0/2057339003 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0f180a1150 0x7f0f180a3600 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:03.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.861+0000 7f0f33d15700 1 --2- 172.21.15.88:0/2057339003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f2c0ffcc0 0x7f0f2c195a90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:03.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.861+0000 7f0f33d15700 1 --2- 172.21.15.88:0/2057339003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f2c100ec0 0x7f0f2c195fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:03.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.861+0000 7f0f33d15700 1 -- 172.21.15.88:0/2057339003 >> 172.21.15.88:0/2057339003 conn(0x7f0f2c0fb2d0 msgr2=0x7f0f2c104420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:03.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.861+0000 7f0f33d15700 1 -- 172.21.15.88:0/2057339003 shutdown_connections 2023-07-14T17:42:03.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:03.861+0000 7f0f33d15700 1 -- 172.21.15.88:0/2057339003 wait complete. 2023-07-14T17:42:04.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:03 smithi178 ceph-mon[129368]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T17:42:04.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:03 smithi088 ceph-mon[127465]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T17:42:04.200 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":78,"num_osds":12,"num_up_osds":12,"osd_up_since":1689356521,"num_in_osds":12,"osd_in_since":1689356503,"num_remapped_pgs":0} 2023-07-14T17:42:04.201 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T17:42:04.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:03 smithi107 ceph-mon[128999]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T17:42:04.754 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:42:05.155 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:04 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2057339003' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T17:42:05.155 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:04 smithi088 ceph-mon[127465]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:05.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:04 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2057339003' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T17:42:05.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:04 smithi178 ceph-mon[129368]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:05.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:04 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2057339003' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T17:42:05.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:04 smithi107 ceph-mon[128999]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:05.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.966+0000 7ff82514c700 1 -- 172.21.15.88:0/4096433341 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff820104120 msgr2=0x7ff82010bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:05.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.966+0000 7ff82514c700 1 --2- 172.21.15.88:0/4096433341 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff820104120 0x7ff82010bca0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff81800b0d0 tx=0x7ff81800b3e0 comp rx=0 tx=0).stop 2023-07-14T17:42:05.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.966+0000 7ff82514c700 1 -- 172.21.15.88:0/4096433341 shutdown_connections 2023-07-14T17:42:05.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.966+0000 7ff82514c700 1 --2- 172.21.15.88:0/4096433341 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff820104120 0x7ff82010bca0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:05.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.966+0000 7ff82514c700 1 --2- 172.21.15.88:0/4096433341 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8200ff5a0 0x7ff8200ff970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:05.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.966+0000 7ff82514c700 1 --2- 172.21.15.88:0/4096433341 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8200fff40 0x7ff8201039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:05.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.966+0000 7ff82514c700 1 -- 172.21.15.88:0/4096433341 >> 172.21.15.88:0/4096433341 conn(0x7ff820075280 msgr2=0x7ff820075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:05.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.966+0000 7ff82514c700 1 -- 172.21.15.88:0/4096433341 shutdown_connections 2023-07-14T17:42:05.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.966+0000 7ff82514c700 1 -- 172.21.15.88:0/4096433341 wait complete. 2023-07-14T17:42:05.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.966+0000 7ff82514c700 1 Processor -- start 2023-07-14T17:42:05.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.967+0000 7ff82514c700 1 -- start start 2023-07-14T17:42:05.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.967+0000 7ff82514c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8200ff5a0 0x7ff820195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:05.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.967+0000 7ff82514c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8200fff40 0x7ff820196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:05.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.967+0000 7ff82514c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff820104120 0x7ff82019a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:05.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.967+0000 7ff82514c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82019aef0 con 0x7ff8200ff5a0 2023-07-14T17:42:05.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.967+0000 7ff82514c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82019b060 con 0x7ff8200fff40 2023-07-14T17:42:05.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.967+0000 7ff82514c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82019b220 con 0x7ff820104120 2023-07-14T17:42:05.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.967+0000 7ff81ed9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8200ff5a0 0x7ff820195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:05.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.967+0000 7ff81ed9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8200ff5a0 0x7ff820195f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41074/0 (socket says 172.21.15.88:41074) 2023-07-14T17:42:05.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.967+0000 7ff81ed9d700 1 -- 172.21.15.88:0/2453932978 learned_addr learned my addr 172.21.15.88:0/2453932978 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:05.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.967+0000 7ff81e59c700 1 --2- 172.21.15.88:0/2453932978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8200fff40 0x7ff820196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:05.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.967+0000 7ff81f59e700 1 --2- 172.21.15.88:0/2453932978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff820104120 0x7ff82019a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:05.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.967+0000 7ff81ed9d700 1 -- 172.21.15.88:0/2453932978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff820104120 msgr2=0x7ff82019a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:05.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.967+0000 7ff81ed9d700 1 --2- 172.21.15.88:0/2453932978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff820104120 0x7ff82019a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:05.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.967+0000 7ff81ed9d700 1 -- 172.21.15.88:0/2453932978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8200fff40 msgr2=0x7ff820196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:05.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.967+0000 7ff81ed9d700 1 --2- 172.21.15.88:0/2453932978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8200fff40 0x7ff820196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:05.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.967+0000 7ff81ed9d700 1 -- 172.21.15.88:0/2453932978 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff818009d00 con 0x7ff8200ff5a0 2023-07-14T17:42:05.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.967+0000 7ff81ed9d700 1 --2- 172.21.15.88:0/2453932978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8200ff5a0 0x7ff820195f30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff81000c9b0 tx=0x7ff81000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:05.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.968+0000 7ff807fff700 1 -- 172.21.15.88:0/2453932978 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff810007b00 con 0x7ff8200ff5a0 2023-07-14T17:42:05.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.968+0000 7ff807fff700 1 -- 172.21.15.88:0/2453932978 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff81000ce50 con 0x7ff8200ff5a0 2023-07-14T17:42:05.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.968+0000 7ff82514c700 1 -- 172.21.15.88:0/2453932978 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff82019b500 con 0x7ff8200ff5a0 2023-07-14T17:42:05.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.968+0000 7ff807fff700 1 -- 172.21.15.88:0/2453932978 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff810018420 con 0x7ff8200ff5a0 2023-07-14T17:42:05.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.968+0000 7ff82514c700 1 -- 172.21.15.88:0/2453932978 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8201a2d60 con 0x7ff8200ff5a0 2023-07-14T17:42:05.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.969+0000 7ff82514c700 1 -- 172.21.15.88:0/2453932978 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff82004e340 con 0x7ff8200ff5a0 2023-07-14T17:42:05.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.970+0000 7ff807fff700 1 -- 172.21.15.88:0/2453932978 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff81000f450 con 0x7ff8200ff5a0 2023-07-14T17:42:05.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.970+0000 7ff807fff700 1 --2- 172.21.15.88:0/2453932978 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff8080a1150 0x7ff8080a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:05.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.970+0000 7ff81e59c700 1 --2- 172.21.15.88:0/2453932978 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff8080a1150 0x7ff8080a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:05.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.970+0000 7ff807fff700 1 -- 172.21.15.88:0/2453932978 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff8100cc3a0 con 0x7ff8200ff5a0 2023-07-14T17:42:05.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.971+0000 7ff81e59c700 1 --2- 172.21.15.88:0/2453932978 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff8080a1150 0x7ff8080a3600 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff820197550 tx=0x7ff81400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:05.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:05.974+0000 7ff807fff700 1 -- 172.21.15.88:0/2453932978 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff810095f90 con 0x7ff8200ff5a0 2023-07-14T17:42:06.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:06.160+0000 7ff82514c700 1 -- 172.21.15.88:0/2453932978 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8201a2f50 con 0x7ff8200ff5a0 2023-07-14T17:42:06.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:06.161+0000 7ff807fff700 1 -- 172.21.15.88:0/2453932978 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+18447 (secure 0 0 0) 0x7ff81000f700 con 0x7ff8200ff5a0 2023-07-14T17:42:06.144 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:06.145 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":78,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:42:02.830643+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:42:06.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:06.164+0000 7ff82514c700 1 -- 172.21.15.88:0/2453932978 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff8080a1150 msgr2=0x7ff8080a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:06.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:06.164+0000 7ff82514c700 1 --2- 172.21.15.88:0/2453932978 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff8080a1150 0x7ff8080a3600 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff820197550 tx=0x7ff81400b410 comp rx=0 tx=0).stop 2023-07-14T17:42:06.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:06.164+0000 7ff82514c700 1 -- 172.21.15.88:0/2453932978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8200ff5a0 msgr2=0x7ff820195f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:06.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:06.164+0000 7ff82514c700 1 --2- 172.21.15.88:0/2453932978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8200ff5a0 0x7ff820195f30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff81000c9b0 tx=0x7ff81000ccc0 comp rx=0 tx=0).stop 2023-07-14T17:42:06.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:06.164+0000 7ff82514c700 1 -- 172.21.15.88:0/2453932978 shutdown_connections 2023-07-14T17:42:06.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:06.164+0000 7ff82514c700 1 --2- 172.21.15.88:0/2453932978 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff820104120 0x7ff82019a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:06.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:06.164+0000 7ff82514c700 1 --2- 172.21.15.88:0/2453932978 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff8080a1150 0x7ff8080a3600 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:06.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:06.164+0000 7ff82514c700 1 --2- 172.21.15.88:0/2453932978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8200ff5a0 0x7ff820195f30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:06.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:06.164+0000 7ff82514c700 1 --2- 172.21.15.88:0/2453932978 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8200fff40 0x7ff820196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:06.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:06.164+0000 7ff82514c700 1 -- 172.21.15.88:0/2453932978 >> 172.21.15.88:0/2453932978 conn(0x7ff820075280 msgr2=0x7ff8200fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:06.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:06.164+0000 7ff82514c700 1 -- 172.21.15.88:0/2453932978 shutdown_connections 2023-07-14T17:42:06.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:06.164+0000 7ff82514c700 1 -- 172.21.15.88:0/2453932978 wait complete. 2023-07-14T17:42:06.509 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T17:38:50.126535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T17:42:06.509 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-07-14T17:42:06.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:06 smithi088 ceph-mon[127465]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:06.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:06 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2453932978' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:06.974 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:06 smithi107 ceph-mon[128999]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:06.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:06 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2453932978' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:07.062 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:42:07.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:06 smithi178 ceph-mon[129368]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:07.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:06 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2453932978' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:08.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.305+0000 7f061def8700 1 -- 172.21.15.88:0/2212285580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06180fe4b0 msgr2=0x7f06180fe880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:08.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.305+0000 7f061def8700 1 --2- 172.21.15.88:0/2212285580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06180fe4b0 0x7f06180fe880 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f06000099c0 tx=0x7f0600009cd0 comp rx=0 tx=0).stop 2023-07-14T17:42:08.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.305+0000 7f061def8700 1 -- 172.21.15.88:0/2212285580 shutdown_connections 2023-07-14T17:42:08.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.305+0000 7f061def8700 1 --2- 172.21.15.88:0/2212285580 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0618105010 0x7f061810cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:08.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.305+0000 7f061def8700 1 --2- 172.21.15.88:0/2212285580 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06180fe4b0 0x7f06180fe880 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:08.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.305+0000 7f061def8700 1 --2- 172.21.15.88:0/2212285580 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06180fee50 0x7f0618104930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:08.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.305+0000 7f061def8700 1 -- 172.21.15.88:0/2212285580 >> 172.21.15.88:0/2212285580 conn(0x7f0618075280 msgr2=0x7f0618075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:08.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.305+0000 7f061def8700 1 -- 172.21.15.88:0/2212285580 shutdown_connections 2023-07-14T17:42:08.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.305+0000 7f061def8700 1 -- 172.21.15.88:0/2212285580 wait complete. 2023-07-14T17:42:08.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.306+0000 7f061def8700 1 Processor -- start 2023-07-14T17:42:08.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.306+0000 7f061def8700 1 -- start start 2023-07-14T17:42:08.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.306+0000 7f061def8700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06180fe4b0 0x7f061819ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:08.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.306+0000 7f061def8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06180fee50 0x7f061819bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:08.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.306+0000 7f061def8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0618105010 0x7f0618195a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:08.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.306+0000 7f061def8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06181960f0 con 0x7f0618105010 2023-07-14T17:42:08.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.306+0000 7f061def8700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0618196260 con 0x7f06180fee50 2023-07-14T17:42:08.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.306+0000 7f061def8700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0618196420 con 0x7f06180fe4b0 2023-07-14T17:42:08.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.306+0000 7f0617fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0618105010 0x7f0618195a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:08.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.306+0000 7f0617fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0618105010 0x7f0618195a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41090/0 (socket says 172.21.15.88:41090) 2023-07-14T17:42:08.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.306+0000 7f0617fff700 1 -- 172.21.15.88:0/422001994 learned_addr learned my addr 172.21.15.88:0/422001994 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:08.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.306+0000 7f06177fe700 1 --2- 172.21.15.88:0/422001994 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06180fe4b0 0x7f061819ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:08.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.306+0000 7f0616ffd700 1 --2- 172.21.15.88:0/422001994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06180fee50 0x7f061819bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:08.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.307+0000 7f0617fff700 1 -- 172.21.15.88:0/422001994 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06180fe4b0 msgr2=0x7f061819ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:08.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.307+0000 7f0617fff700 1 --2- 172.21.15.88:0/422001994 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06180fe4b0 0x7f061819ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:08.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.307+0000 7f0617fff700 1 -- 172.21.15.88:0/422001994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06180fee50 msgr2=0x7f061819bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:08.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.307+0000 7f0617fff700 1 --2- 172.21.15.88:0/422001994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06180fee50 0x7f061819bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:08.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.307+0000 7f0617fff700 1 -- 172.21.15.88:0/422001994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06000095f0 con 0x7f0618105010 2023-07-14T17:42:08.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.307+0000 7f0617fff700 1 --2- 172.21.15.88:0/422001994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0618105010 0x7f0618195a80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f061000b970 tx=0x7f061000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:08.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.307+0000 7f0614ff9700 1 -- 172.21.15.88:0/422001994 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061000d860 con 0x7f0618105010 2023-07-14T17:42:08.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.307+0000 7f0614ff9700 1 -- 172.21.15.88:0/422001994 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f061000de50 con 0x7f0618105010 2023-07-14T17:42:08.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.307+0000 7f061def8700 1 -- 172.21.15.88:0/422001994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0618196700 con 0x7f0618105010 2023-07-14T17:42:08.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.307+0000 7f0614ff9700 1 -- 172.21.15.88:0/422001994 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0610017400 con 0x7f0618105010 2023-07-14T17:42:08.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.307+0000 7f061def8700 1 -- 172.21.15.88:0/422001994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06181a29f0 con 0x7f0618105010 2023-07-14T17:42:08.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.308+0000 7f061def8700 1 -- 172.21.15.88:0/422001994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f061804e340 con 0x7f0618105010 2023-07-14T17:42:08.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.309+0000 7f0614ff9700 1 -- 172.21.15.88:0/422001994 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f061000d9c0 con 0x7f0618105010 2023-07-14T17:42:08.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.310+0000 7f0614ff9700 1 --2- 172.21.15.88:0/422001994 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f06040a1150 0x7f06040a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:08.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.310+0000 7f06177fe700 1 --2- 172.21.15.88:0/422001994 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f06040a1150 0x7f06040a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:08.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.310+0000 7f0614ff9700 1 -- 172.21.15.88:0/422001994 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0610006e40 con 0x7f0618105010 2023-07-14T17:42:08.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.310+0000 7f06177fe700 1 --2- 172.21.15.88:0/422001994 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f06040a1150 0x7f06040a3600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f060000b580 tx=0x7f0600009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:08.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.313+0000 7f0614ff9700 1 -- 172.21.15.88:0/422001994 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0610095dc0 con 0x7f0618105010 2023-07-14T17:42:08.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.502+0000 7f061def8700 1 -- 172.21.15.88:0/422001994 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0618196f90 con 0x7f0618105010 2023-07-14T17:42:08.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.503+0000 7f0614ff9700 1 -- 172.21.15.88:0/422001994 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f06100993e0 con 0x7f0618105010 2023-07-14T17:42:08.486 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-07-14T17:42:08.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.505+0000 7f061def8700 1 -- 172.21.15.88:0/422001994 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f06040a1150 msgr2=0x7f06040a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:08.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.505+0000 7f061def8700 1 --2- 172.21.15.88:0/422001994 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f06040a1150 0x7f06040a3600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f060000b580 tx=0x7f0600009f10 comp rx=0 tx=0).stop 2023-07-14T17:42:08.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.505+0000 7f061def8700 1 -- 172.21.15.88:0/422001994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0618105010 msgr2=0x7f0618195a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:08.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.505+0000 7f061def8700 1 --2- 172.21.15.88:0/422001994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0618105010 0x7f0618195a80 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f061000b970 tx=0x7f061000bc80 comp rx=0 tx=0).stop 2023-07-14T17:42:08.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.505+0000 7f061def8700 1 -- 172.21.15.88:0/422001994 shutdown_connections 2023-07-14T17:42:08.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.505+0000 7f061def8700 1 --2- 172.21.15.88:0/422001994 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f06040a1150 0x7f06040a3600 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:08.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.505+0000 7f061def8700 1 --2- 172.21.15.88:0/422001994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0618105010 0x7f0618195a80 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:08.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.505+0000 7f061def8700 1 --2- 172.21.15.88:0/422001994 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06180fe4b0 0x7f061819ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:08.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.505+0000 7f061def8700 1 --2- 172.21.15.88:0/422001994 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06180fee50 0x7f061819bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:08.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.505+0000 7f061def8700 1 -- 172.21.15.88:0/422001994 >> 172.21.15.88:0/422001994 conn(0x7f0618075280 msgr2=0x7f06180fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:08.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.505+0000 7f061def8700 1 -- 172.21.15.88:0/422001994 shutdown_connections 2023-07-14T17:42:08.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:08.505+0000 7f061def8700 1 -- 172.21.15.88:0/422001994 wait complete. 2023-07-14T17:42:08.808 INFO:tasks.cephadm:Adding mds.a on smithi088 2023-07-14T17:42:08.808 INFO:tasks.cephadm:Adding mds.d on smithi088 2023-07-14T17:42:08.808 INFO:tasks.cephadm:Adding mds.g on smithi088 2023-07-14T17:42:08.809 INFO:tasks.cephadm:Adding mds.j on smithi088 2023-07-14T17:42:08.809 INFO:tasks.cephadm:Adding mds.b on smithi107 2023-07-14T17:42:08.809 INFO:tasks.cephadm:Adding mds.e on smithi107 2023-07-14T17:42:08.809 INFO:tasks.cephadm:Adding mds.h on smithi107 2023-07-14T17:42:08.809 INFO:tasks.cephadm:Adding mds.k on smithi107 2023-07-14T17:42:08.810 INFO:tasks.cephadm:Adding mds.c on smithi178 2023-07-14T17:42:08.810 INFO:tasks.cephadm:Adding mds.f on smithi178 2023-07-14T17:42:08.810 INFO:tasks.cephadm:Adding mds.i on smithi178 2023-07-14T17:42:08.810 INFO:tasks.cephadm:Adding mds.l on smithi178 2023-07-14T17:42:08.811 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph orch apply mds all '12;smithi088=a;smithi088=d;smithi088=g;smithi088=j;smithi107=b;smithi107=e;smithi107=h;smithi107=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l' 2023-07-14T17:42:09.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:08 smithi178 ceph-mon[129368]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:09.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:08 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/422001994' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:42:09.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:08 smithi088 ceph-mon[127465]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:09.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:08 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/422001994' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:42:09.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:08 smithi107 ceph-mon[128999]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:09.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:08 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/422001994' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:42:09.378 INFO:teuthology.orchestra.run.smithi178.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T17:42:10.709 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.724+0000 7ff448f26700 1 -- 172.21.15.178:0/1579919958 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4440fee10 msgr2=0x7ff4440ff280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:10.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.724+0000 7ff448f26700 1 --2- 172.21.15.178:0/1579919958 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4440fee10 0x7ff4440ff280 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff438009990 tx=0x7ff438009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:10.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.725+0000 7ff448f26700 1 -- 172.21.15.178:0/1579919958 shutdown_connections 2023-07-14T17:42:10.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.725+0000 7ff448f26700 1 --2- 172.21.15.178:0/1579919958 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4440ff7c0 0x7ff444107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:10.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.725+0000 7ff448f26700 1 --2- 172.21.15.178:0/1579919958 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff444100240 0x7ff444100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:10.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.725+0000 7ff448f26700 1 --2- 172.21.15.178:0/1579919958 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4440fee10 0x7ff4440ff280 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:10.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.725+0000 7ff448f26700 1 -- 172.21.15.178:0/1579919958 >> 172.21.15.178:0/1579919958 conn(0x7ff4440f9990 msgr2=0x7ff4440fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:10.710 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.725+0000 7ff448f26700 1 -- 172.21.15.178:0/1579919958 shutdown_connections 2023-07-14T17:42:10.711 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.725+0000 7ff448f26700 1 -- 172.21.15.178:0/1579919958 wait complete. 2023-07-14T17:42:10.711 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.725+0000 7ff448f26700 1 Processor -- start 2023-07-14T17:42:10.711 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.725+0000 7ff448f26700 1 -- start start 2023-07-14T17:42:10.711 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff448f26700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4440fee10 0x7ff4441958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:10.711 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff448f26700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4440ff7c0 0x7ff444195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:10.711 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff448f26700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff444100240 0x7ff44419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:10.711 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff448f26700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44419d470 con 0x7ff444100240 2023-07-14T17:42:10.711 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff448f26700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44419d5b0 con 0x7ff4440ff7c0 2023-07-14T17:42:10.712 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff448f26700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44419d740 con 0x7ff4440fee10 2023-07-14T17:42:10.712 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff44259c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4440fee10 0x7ff4441958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:10.712 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff44259c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4440fee10 0x7ff4441958e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.178:55126/0 (socket says 172.21.15.178:55126) 2023-07-14T17:42:10.712 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff44259c700 1 -- 172.21.15.178:0/2997334680 learned_addr learned my addr 172.21.15.178:0/2997334680 (peer_addr_for_me v2:172.21.15.178:0/0) 2023-07-14T17:42:10.712 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff441d9b700 1 --2- 172.21.15.178:0/2997334680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4440ff7c0 0x7ff444195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:10.712 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff442d9d700 1 --2- 172.21.15.178:0/2997334680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff444100240 0x7ff44419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:10.712 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff44259c700 1 -- 172.21.15.178:0/2997334680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4440ff7c0 msgr2=0x7ff444195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:10.713 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff44259c700 1 --2- 172.21.15.178:0/2997334680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4440ff7c0 0x7ff444195e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:10.713 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff44259c700 1 -- 172.21.15.178:0/2997334680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff444100240 msgr2=0x7ff44419cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:10.713 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff44259c700 1 --2- 172.21.15.178:0/2997334680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff444100240 0x7ff44419cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:10.713 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff44259c700 1 -- 172.21.15.178:0/2997334680 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4380095f0 con 0x7ff4440fee10 2023-07-14T17:42:10.713 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff44259c700 1 --2- 172.21.15.178:0/2997334680 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4440fee10 0x7ff4441958e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff43400c950 tx=0x7ff43400cd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:10.713 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff4337fe700 1 -- 172.21.15.178:0/2997334680 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff434007ac0 con 0x7ff4440fee10 2023-07-14T17:42:10.713 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff4337fe700 1 -- 172.21.15.178:0/2997334680 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff434004cc0 con 0x7ff4440fee10 2023-07-14T17:42:10.714 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff4337fe700 1 -- 172.21.15.178:0/2997334680 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4340054e0 con 0x7ff4440fee10 2023-07-14T17:42:10.714 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff448f26700 1 -- 172.21.15.178:0/2997334680 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff44419da20 con 0x7ff4440fee10 2023-07-14T17:42:10.714 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.726+0000 7ff448f26700 1 -- 172.21.15.178:0/2997334680 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff44419ded0 con 0x7ff4440fee10 2023-07-14T17:42:10.715 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.727+0000 7ff448f26700 1 -- 172.21.15.178:0/2997334680 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff44410b390 con 0x7ff4440fee10 2023-07-14T17:42:10.715 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.728+0000 7ff4337fe700 1 -- 172.21.15.178:0/2997334680 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff434004750 con 0x7ff4440fee10 2023-07-14T17:42:10.715 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.729+0000 7ff4337fe700 1 --2- 172.21.15.178:0/2997334680 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff42c0a1150 0x7ff42c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:10.715 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.729+0000 7ff4337fe700 1 -- 172.21.15.178:0/2997334680 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff4340cc480 con 0x7ff4440fee10 2023-07-14T17:42:10.715 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.729+0000 7ff441d9b700 1 --2- 172.21.15.178:0/2997334680 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff42c0a1150 0x7ff42c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:10.715 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.730+0000 7ff441d9b700 1 --2- 172.21.15.178:0/2997334680 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff42c0a1150 0x7ff42c0a3600 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff43800b5c0 tx=0x7ff438005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:10.716 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.732+0000 7ff4337fe700 1 -- 172.21.15.178:0/2997334680 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff434096070 con 0x7ff4440fee10 2023-07-14T17:42:10.910 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.928+0000 7ff448f26700 1 -- 172.21.15.178:0/2997334680 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi088=a;smithi088=d;smithi088=g;smithi088=j;smithi107=b;smithi107=e;smithi107=h;smithi107=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l", "target": ["mon-mgr", ""]}) v1 -- 0x7ff444060b80 con 0x7ff42c0a1150 2023-07-14T17:42:10.918 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.936+0000 7ff4337fe700 1 -- 172.21.15.178:0/2997334680 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7ff444060b80 con 0x7ff42c0a1150 2023-07-14T17:42:10.919 INFO:teuthology.orchestra.run.smithi178.stdout:Scheduled mds.all update... 2023-07-14T17:42:10.920 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.938+0000 7ff448f26700 1 -- 172.21.15.178:0/2997334680 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff42c0a1150 msgr2=0x7ff42c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:10.921 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.938+0000 7ff448f26700 1 --2- 172.21.15.178:0/2997334680 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff42c0a1150 0x7ff42c0a3600 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff43800b5c0 tx=0x7ff438005fb0 comp rx=0 tx=0).stop 2023-07-14T17:42:10.921 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.938+0000 7ff448f26700 1 -- 172.21.15.178:0/2997334680 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4440fee10 msgr2=0x7ff4441958e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:10.921 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.938+0000 7ff448f26700 1 --2- 172.21.15.178:0/2997334680 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4440fee10 0x7ff4441958e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff43400c950 tx=0x7ff43400cd10 comp rx=0 tx=0).stop 2023-07-14T17:42:10.921 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.938+0000 7ff448f26700 1 -- 172.21.15.178:0/2997334680 shutdown_connections 2023-07-14T17:42:10.921 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.938+0000 7ff448f26700 1 --2- 172.21.15.178:0/2997334680 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff42c0a1150 0x7ff42c0a3600 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:10.921 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.938+0000 7ff448f26700 1 --2- 172.21.15.178:0/2997334680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff444100240 0x7ff44419cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:10.921 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.938+0000 7ff448f26700 1 --2- 172.21.15.178:0/2997334680 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4440fee10 0x7ff4441958e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:10.922 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.938+0000 7ff448f26700 1 --2- 172.21.15.178:0/2997334680 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4440ff7c0 0x7ff444195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:10.922 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.938+0000 7ff448f26700 1 -- 172.21.15.178:0/2997334680 >> 172.21.15.178:0/2997334680 conn(0x7ff4440f9990 msgr2=0x7ff4440fbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:10.922 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.938+0000 7ff448f26700 1 -- 172.21.15.178:0/2997334680 shutdown_connections 2023-07-14T17:42:10.922 INFO:teuthology.orchestra.run.smithi178.stderr:2023-07-14T17:42:10.938+0000 7ff448f26700 1 -- 172.21.15.178:0/2997334680 wait complete. 2023-07-14T17:42:11.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:10 smithi178 ceph-mon[129368]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:11.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:10 smithi088 ceph-mon[127465]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:11.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:10 smithi107 ceph-mon[128999]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:11.325 DEBUG:teuthology.orchestra.run.smithi088:mds.a> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.a.service 2023-07-14T17:42:11.328 DEBUG:teuthology.orchestra.run.smithi088:mds.d> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.d.service 2023-07-14T17:42:11.364 INFO:journalctl@ceph.mds.a.smithi088.stdout:-- Logs begin at Fri 2023-07-14 17:14:35 UTC. -- 2023-07-14T17:42:11.365 DEBUG:teuthology.orchestra.run.smithi088:mds.g> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.g.service 2023-07-14T17:42:11.394 INFO:journalctl@ceph.mds.d.smithi088.stdout:-- Logs begin at Fri 2023-07-14 17:14:35 UTC. -- 2023-07-14T17:42:11.395 DEBUG:teuthology.orchestra.run.smithi088:mds.j> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.j.service 2023-07-14T17:42:11.422 INFO:journalctl@ceph.mds.g.smithi088.stdout:-- Logs begin at Fri 2023-07-14 17:14:35 UTC. -- 2023-07-14T17:42:11.423 DEBUG:teuthology.orchestra.run.smithi107:mds.b> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.b.service 2023-07-14T17:42:11.426 DEBUG:teuthology.orchestra.run.smithi107:mds.e> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.e.service 2023-07-14T17:42:11.449 INFO:journalctl@ceph.mds.j.smithi088.stdout:-- Logs begin at Fri 2023-07-14 17:14:35 UTC. -- 2023-07-14T17:42:11.463 INFO:journalctl@ceph.mds.b.smithi107.stdout:-- Logs begin at Fri 2023-07-14 17:14:00 UTC. -- 2023-07-14T17:42:11.464 DEBUG:teuthology.orchestra.run.smithi107:mds.h> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.h.service 2023-07-14T17:42:11.493 INFO:journalctl@ceph.mds.e.smithi107.stdout:-- Logs begin at Fri 2023-07-14 17:14:00 UTC. -- 2023-07-14T17:42:11.495 DEBUG:teuthology.orchestra.run.smithi107:mds.k> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.k.service 2023-07-14T17:42:11.522 INFO:journalctl@ceph.mds.h.smithi107.stdout:-- Logs begin at Fri 2023-07-14 17:14:00 UTC. -- 2023-07-14T17:42:11.522 DEBUG:teuthology.orchestra.run.smithi178:mds.c> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.c.service 2023-07-14T17:42:11.525 DEBUG:teuthology.orchestra.run.smithi178:mds.f> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.f.service 2023-07-14T17:42:11.549 INFO:journalctl@ceph.mds.k.smithi107.stdout:-- Logs begin at Fri 2023-07-14 17:14:00 UTC. -- 2023-07-14T17:42:11.552 INFO:journalctl@ceph.mds.c.smithi178.stdout:-- Logs begin at Fri 2023-07-14 17:14:18 UTC. -- 2023-07-14T17:42:11.554 DEBUG:teuthology.orchestra.run.smithi178:mds.i> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.i.service 2023-07-14T17:42:11.579 INFO:journalctl@ceph.mds.f.smithi178.stdout:-- Logs begin at Fri 2023-07-14 17:14:18 UTC. -- 2023-07-14T17:42:11.580 DEBUG:teuthology.orchestra.run.smithi178:mds.l> sudo journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.l.service 2023-07-14T17:42:11.607 INFO:journalctl@ceph.mds.i.smithi178.stdout:-- Logs begin at Fri 2023-07-14 17:14:18 UTC. -- 2023-07-14T17:42:11.608 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-07-14T17:42:11.609 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:42:11.634 INFO:journalctl@ceph.mds.l.smithi178.stdout:-- Logs begin at Fri 2023-07-14 17:14:18 UTC. -- 2023-07-14T17:42:11.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.761+0000 7fb33b77b700 1 -- 172.21.15.88:0/3765257229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3340ffc30 msgr2=0x7fb334100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:11.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.761+0000 7fb33b77b700 1 --2- 172.21.15.88:0/3765257229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3340ffc30 0x7fb334100040 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb330009990 tx=0x7fb330009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:11.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.761+0000 7fb33b77b700 1 -- 172.21.15.88:0/3765257229 shutdown_connections 2023-07-14T17:42:11.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.761+0000 7fb33b77b700 1 --2- 172.21.15.88:0/3765257229 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3341017b0 0x7fb334108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:11.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.761+0000 7fb33b77b700 1 --2- 172.21.15.88:0/3765257229 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb3340ffc30 0x7fb334100040 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:11.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.761+0000 7fb33b77b700 1 --2- 172.21.15.88:0/3765257229 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb334100e20 0x7fb334101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:11.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.761+0000 7fb33b77b700 1 -- 172.21.15.88:0/3765257229 >> 172.21.15.88:0/3765257229 conn(0x7fb3340fb210 msgr2=0x7fb3340fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:11.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.761+0000 7fb33b77b700 1 -- 172.21.15.88:0/3765257229 shutdown_connections 2023-07-14T17:42:11.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.761+0000 7fb33b77b700 1 -- 172.21.15.88:0/3765257229 wait complete. 2023-07-14T17:42:11.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.761+0000 7fb33b77b700 1 Processor -- start 2023-07-14T17:42:11.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.761+0000 7fb33b77b700 1 -- start start 2023-07-14T17:42:11.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.761+0000 7fb33b77b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3340ffc30 0x7fb334195840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:11.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.761+0000 7fb33b77b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb334100e20 0x7fb334195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:11.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.761+0000 7fb33b77b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3341017b0 0x7fb33419cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:11.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb33b77b700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33419d3c0 con 0x7fb334100e20 2023-07-14T17:42:11.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb33b77b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33419d500 con 0x7fb3341017b0 2023-07-14T17:42:11.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb33b77b700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33419d690 con 0x7fb3340ffc30 2023-07-14T17:42:11.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb338d16700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb334100e20 0x7fb334195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:11.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb338d16700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb334100e20 0x7fb334195d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41102/0 (socket says 172.21.15.88:41102) 2023-07-14T17:42:11.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb338d16700 1 -- 172.21.15.88:0/2907831292 learned_addr learned my addr 172.21.15.88:0/2907831292 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:11.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb339517700 1 --2- 172.21.15.88:0/2907831292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3340ffc30 0x7fb334195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:11.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb339d18700 1 --2- 172.21.15.88:0/2907831292 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3341017b0 0x7fb33419cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:11.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb338d16700 1 -- 172.21.15.88:0/2907831292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3340ffc30 msgr2=0x7fb334195840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:11.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb338d16700 1 --2- 172.21.15.88:0/2907831292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3340ffc30 0x7fb334195840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:11.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb338d16700 1 -- 172.21.15.88:0/2907831292 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3341017b0 msgr2=0x7fb33419cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:11.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb338d16700 1 --2- 172.21.15.88:0/2907831292 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3341017b0 0x7fb33419cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:11.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb338d16700 1 -- 172.21.15.88:0/2907831292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3300095f0 con 0x7fb334100e20 2023-07-14T17:42:11.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb339d18700 1 --2- 172.21.15.88:0/2907831292 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3341017b0 0x7fb33419cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:42:11.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb338d16700 1 --2- 172.21.15.88:0/2907831292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb334100e20 0x7fb334195d80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb32800e9a0 tx=0x7fb32800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:11.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb3267fc700 1 -- 172.21.15.88:0/2907831292 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb328009790 con 0x7fb334100e20 2023-07-14T17:42:11.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb3267fc700 1 -- 172.21.15.88:0/2907831292 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb32800cbd0 con 0x7fb334100e20 2023-07-14T17:42:11.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb33b77b700 1 -- 172.21.15.88:0/2907831292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb33419d970 con 0x7fb334100e20 2023-07-14T17:42:11.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb3267fc700 1 -- 172.21.15.88:0/2907831292 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb328018680 con 0x7fb334100e20 2023-07-14T17:42:11.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.762+0000 7fb33b77b700 1 -- 172.21.15.88:0/2907831292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb33419de80 con 0x7fb334100e20 2023-07-14T17:42:11.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.763+0000 7fb33b77b700 1 -- 172.21.15.88:0/2907831292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb334106bb0 con 0x7fb334100e20 2023-07-14T17:42:11.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.764+0000 7fb3267fc700 1 -- 172.21.15.88:0/2907831292 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb32800cd40 con 0x7fb334100e20 2023-07-14T17:42:11.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.765+0000 7fb3267fc700 1 --2- 172.21.15.88:0/2907831292 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb3200a11a0 0x7fb3200a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:11.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.765+0000 7fb339517700 1 --2- 172.21.15.88:0/2907831292 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb3200a11a0 0x7fb3200a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:11.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.765+0000 7fb3267fc700 1 -- 172.21.15.88:0/2907831292 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb328014070 con 0x7fb334100e20 2023-07-14T17:42:11.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.765+0000 7fb339517700 1 --2- 172.21.15.88:0/2907831292 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb3200a11a0 0x7fb3200a3650 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb330006010 tx=0x7fb33000b2a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:11.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.768+0000 7fb3267fc700 1 -- 172.21.15.88:0/2907831292 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb328096620 con 0x7fb334100e20 2023-07-14T17:42:11.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.957+0000 7fb33b77b700 1 -- 172.21.15.88:0/2907831292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb33402cab0 con 0x7fb334100e20 2023-07-14T17:42:11.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.958+0000 7fb3267fc700 1 -- 172.21.15.88:0/2907831292 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+18447 (secure 0 0 0) 0x7fb328099c40 con 0x7fb334100e20 2023-07-14T17:42:11.941 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:11.942 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":78,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:42:02.830643+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:42:11.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.960+0000 7fb33b77b700 1 -- 172.21.15.88:0/2907831292 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb3200a11a0 msgr2=0x7fb3200a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:11.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.960+0000 7fb33b77b700 1 --2- 172.21.15.88:0/2907831292 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb3200a11a0 0x7fb3200a3650 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb330006010 tx=0x7fb33000b2a0 comp rx=0 tx=0).stop 2023-07-14T17:42:11.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.960+0000 7fb33b77b700 1 -- 172.21.15.88:0/2907831292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb334100e20 msgr2=0x7fb334195d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:11.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.960+0000 7fb33b77b700 1 --2- 172.21.15.88:0/2907831292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb334100e20 0x7fb334195d80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb32800e9a0 tx=0x7fb32800ecb0 comp rx=0 tx=0).stop 2023-07-14T17:42:11.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.960+0000 7fb33b77b700 1 -- 172.21.15.88:0/2907831292 shutdown_connections 2023-07-14T17:42:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.960+0000 7fb33b77b700 1 --2- 172.21.15.88:0/2907831292 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb3341017b0 0x7fb33419cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.960+0000 7fb33b77b700 1 --2- 172.21.15.88:0/2907831292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb3340ffc30 0x7fb334195840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.960+0000 7fb33b77b700 1 --2- 172.21.15.88:0/2907831292 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb3200a11a0 0x7fb3200a3650 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.960+0000 7fb33b77b700 1 --2- 172.21.15.88:0/2907831292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb334100e20 0x7fb334195d80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.960+0000 7fb33b77b700 1 -- 172.21.15.88:0/2907831292 >> 172.21.15.88:0/2907831292 conn(0x7fb3340fb210 msgr2=0x7fb3340fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.960+0000 7fb33b77b700 1 -- 172.21.15.88:0/2907831292 shutdown_connections 2023-07-14T17:42:11.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:11.960+0000 7fb33b77b700 1 -- 172.21.15.88:0/2907831292 wait complete. 2023-07-14T17:42:11.955 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T17:38:50.126535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T17:42:11.956 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T17:42:12.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.097+0000 7fbc38f9f700 1 -- 172.21.15.88:0/4041409138 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc341058d0 msgr2=0x7fbc34107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:12.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.097+0000 7fbc38f9f700 1 --2- 172.21.15.88:0/4041409138 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc341058d0 0x7fbc34107d30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c00b0a0 tx=0x7fbc2c00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.097+0000 7fbc38f9f700 1 -- 172.21.15.88:0/4041409138 shutdown_connections 2023-07-14T17:42:12.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.097+0000 7fbc38f9f700 1 --2- 172.21.15.88:0/4041409138 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc341058d0 0x7fbc34107d30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.097+0000 7fbc38f9f700 1 --2- 172.21.15.88:0/4041409138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc34100690 0x7fbc34102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.097+0000 7fbc38f9f700 1 --2- 172.21.15.88:0/4041409138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc34102fb0 0x7fbc34105390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.097+0000 7fbc38f9f700 1 -- 172.21.15.88:0/4041409138 >> 172.21.15.88:0/4041409138 conn(0x7fbc340fa130 msgr2=0x7fbc340fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:12.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.097+0000 7fbc38f9f700 1 -- 172.21.15.88:0/4041409138 shutdown_connections 2023-07-14T17:42:12.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.097+0000 7fbc38f9f700 1 -- 172.21.15.88:0/4041409138 wait complete. 2023-07-14T17:42:12.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.097+0000 7fbc38f9f700 1 Processor -- start 2023-07-14T17:42:12.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.097+0000 7fbc38f9f700 1 -- start start 2023-07-14T17:42:12.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc38f9f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc34100690 0x7fbc341958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:12.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc38f9f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc34102fb0 0x7fbc34195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:12.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc38f9f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc341058d0 0x7fbc3419ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:12.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc38f9f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3419d430 con 0x7fbc34100690 2023-07-14T17:42:12.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc38f9f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3419d570 con 0x7fbc341058d0 2023-07-14T17:42:12.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc38f9f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3419d700 con 0x7fbc34102fb0 2023-07-14T17:42:12.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc3259c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc34100690 0x7fbc341958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:12.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc3259c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc34100690 0x7fbc341958b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41114/0 (socket says 172.21.15.88:41114) 2023-07-14T17:42:12.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc3259c700 1 -- 172.21.15.88:0/3379038062 learned_addr learned my addr 172.21.15.88:0/3379038062 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:12.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc32d9d700 1 --2- 172.21.15.88:0/3379038062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc341058d0 0x7fbc3419ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:12.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc31d9b700 1 --2- 172.21.15.88:0/3379038062 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc34102fb0 0x7fbc34195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:12.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc3259c700 1 -- 172.21.15.88:0/3379038062 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc34102fb0 msgr2=0x7fbc34195df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:12.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc3259c700 1 --2- 172.21.15.88:0/3379038062 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc34102fb0 0x7fbc34195df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc3259c700 1 -- 172.21.15.88:0/3379038062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc341058d0 msgr2=0x7fbc3419ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:12.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc3259c700 1 --2- 172.21.15.88:0/3379038062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc341058d0 0x7fbc3419ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc3259c700 1 -- 172.21.15.88:0/3379038062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c009d00 con 0x7fbc34100690 2023-07-14T17:42:12.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc32d9d700 1 --2- 172.21.15.88:0/3379038062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc341058d0 0x7fbc3419ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:42:12.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc3259c700 1 --2- 172.21.15.88:0/3379038062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc34100690 0x7fbc341958b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800e9a0 tx=0x7fbc2800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:12.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc237fe700 1 -- 172.21.15.88:0/3379038062 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28009790 con 0x7fbc34100690 2023-07-14T17:42:12.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc38f9f700 1 -- 172.21.15.88:0/3379038062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc3419d9e0 con 0x7fbc34100690 2023-07-14T17:42:12.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc237fe700 1 -- 172.21.15.88:0/3379038062 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fbc2800cbd0 con 0x7fbc34100690 2023-07-14T17:42:12.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc237fe700 1 -- 172.21.15.88:0/3379038062 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28018680 con 0x7fbc34100690 2023-07-14T17:42:12.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.098+0000 7fbc38f9f700 1 -- 172.21.15.88:0/3379038062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc3419df40 con 0x7fbc34100690 2023-07-14T17:42:12.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.099+0000 7fbc38f9f700 1 -- 172.21.15.88:0/3379038062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc340fbca0 con 0x7fbc34100690 2023-07-14T17:42:12.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.100+0000 7fbc237fe700 1 -- 172.21.15.88:0/3379038062 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbc2800cd40 con 0x7fbc34100690 2023-07-14T17:42:12.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.101+0000 7fbc237fe700 1 --2- 172.21.15.88:0/3379038062 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbc1c0a1150 0x7fbc1c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:12.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.101+0000 7fbc31d9b700 1 --2- 172.21.15.88:0/3379038062 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbc1c0a1150 0x7fbc1c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:12.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.101+0000 7fbc237fe700 1 -- 172.21.15.88:0/3379038062 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbc28014070 con 0x7fbc34100690 2023-07-14T17:42:12.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.101+0000 7fbc31d9b700 1 --2- 172.21.15.88:0/3379038062 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbc1c0a1150 0x7fbc1c0a3600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400ac50 tx=0x7fbc2400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:12.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.104+0000 7fbc237fe700 1 -- 172.21.15.88:0/3379038062 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbc280966c0 con 0x7fbc34100690 2023-07-14T17:42:12.158 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:11 smithi178 ceph-mon[129368]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi088=a;smithi088=d;smithi088=g;smithi088=j;smithi107=b;smithi107=e;smithi107=h;smithi107=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:42:12.159 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:11 smithi178 ceph-mon[129368]: Saving service mds.all spec with placement smithi088=a;smithi088=d;smithi088=g;smithi088=j;smithi107=b;smithi107=e;smithi107=h;smithi107=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l;count:12 2023-07-14T17:42:12.159 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:11 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:12.159 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:11 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:42:12.159 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:11 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:12.159 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:11 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:42:12.160 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:11 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:12.160 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:11 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:12.160 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:11 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:12.160 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:11 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:12.160 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:11 smithi178 ceph-mon[129368]: Deploying daemon mds.l on smithi178 2023-07-14T17:42:12.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:11 smithi088 ceph-mon[127465]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi088=a;smithi088=d;smithi088=g;smithi088=j;smithi107=b;smithi107=e;smithi107=h;smithi107=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:42:12.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:11 smithi088 ceph-mon[127465]: Saving service mds.all spec with placement smithi088=a;smithi088=d;smithi088=g;smithi088=j;smithi107=b;smithi107=e;smithi107=h;smithi107=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l;count:12 2023-07-14T17:42:12.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:11 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:12.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:11 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:42:12.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:11 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:12.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:11 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:42:12.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:11 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:12.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:11 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:12.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:11 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:12.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:11 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:12.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:11 smithi088 ceph-mon[127465]: Deploying daemon mds.l on smithi178 2023-07-14T17:42:12.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:11 smithi107 ceph-mon[128999]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi088=a;smithi088=d;smithi088=g;smithi088=j;smithi107=b;smithi107=e;smithi107=h;smithi107=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:42:12.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:11 smithi107 ceph-mon[128999]: Saving service mds.all spec with placement smithi088=a;smithi088=d;smithi088=g;smithi088=j;smithi107=b;smithi107=e;smithi107=h;smithi107=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l;count:12 2023-07-14T17:42:12.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:11 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:12.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:11 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:42:12.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:11 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:12.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:11 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:42:12.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:11 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:12.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:11 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:12.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:11 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:12.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:11 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:12.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:11 smithi107 ceph-mon[128999]: Deploying daemon mds.l on smithi178 2023-07-14T17:42:12.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.293+0000 7fbc38f9f700 1 -- 172.21.15.88:0/3379038062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbc3402ca90 con 0x7fbc34100690 2023-07-14T17:42:12.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.293+0000 7fbc237fe700 1 -- 172.21.15.88:0/3379038062 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbc28099ce0 con 0x7fbc34100690 2023-07-14T17:42:12.276 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-07-14T17:42:12.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.296+0000 7fbc38f9f700 1 -- 172.21.15.88:0/3379038062 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbc1c0a1150 msgr2=0x7fbc1c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:12.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.296+0000 7fbc38f9f700 1 --2- 172.21.15.88:0/3379038062 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbc1c0a1150 0x7fbc1c0a3600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400ac50 tx=0x7fbc2400a380 comp rx=0 tx=0).stop 2023-07-14T17:42:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.296+0000 7fbc38f9f700 1 -- 172.21.15.88:0/3379038062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc34100690 msgr2=0x7fbc341958b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.296+0000 7fbc38f9f700 1 --2- 172.21.15.88:0/3379038062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc34100690 0x7fbc341958b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800e9a0 tx=0x7fbc2800ecb0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.296+0000 7fbc38f9f700 1 -- 172.21.15.88:0/3379038062 shutdown_connections 2023-07-14T17:42:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.296+0000 7fbc38f9f700 1 --2- 172.21.15.88:0/3379038062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc341058d0 0x7fbc3419ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.296+0000 7fbc38f9f700 1 --2- 172.21.15.88:0/3379038062 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbc1c0a1150 0x7fbc1c0a3600 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.279 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.296+0000 7fbc38f9f700 1 --2- 172.21.15.88:0/3379038062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc34100690 0x7fbc341958b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.296+0000 7fbc38f9f700 1 --2- 172.21.15.88:0/3379038062 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc34102fb0 0x7fbc34195df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.296+0000 7fbc38f9f700 1 -- 172.21.15.88:0/3379038062 >> 172.21.15.88:0/3379038062 conn(0x7fbc340fa130 msgr2=0x7fbc34101330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:12.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.296+0000 7fbc38f9f700 1 -- 172.21.15.88:0/3379038062 shutdown_connections 2023-07-14T17:42:12.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.296+0000 7fbc38f9f700 1 -- 172.21.15.88:0/3379038062 wait complete. 2023-07-14T17:42:12.290 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:42:12.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.428+0000 7f5406d45700 1 -- 172.21.15.88:0/3049203984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54000ffe00 msgr2=0x7f54001066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:12.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.428+0000 7f5406d45700 1 --2- 172.21.15.88:0/3049203984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54000ffe00 0x7f54001066c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f53f800b0a0 tx=0x7f53f800b3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.428+0000 7f5406d45700 1 -- 172.21.15.88:0/3049203984 shutdown_connections 2023-07-14T17:42:12.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.428+0000 7f5406d45700 1 --2- 172.21.15.88:0/3049203984 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54000ffe00 0x7f54001066c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.428+0000 7f5406d45700 1 --2- 172.21.15.88:0/3049203984 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54000fe280 0x7f54000fe690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.415 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.428+0000 7f5406d45700 1 --2- 172.21.15.88:0/3049203984 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54000ff470 0x7f54000ff8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.428+0000 7f5406d45700 1 -- 172.21.15.88:0/3049203984 >> 172.21.15.88:0/3049203984 conn(0x7f54000f98a0 msgr2=0x7f54000fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:12.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.428+0000 7f5406d45700 1 -- 172.21.15.88:0/3049203984 shutdown_connections 2023-07-14T17:42:12.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.428+0000 7f5406d45700 1 -- 172.21.15.88:0/3049203984 wait complete. 2023-07-14T17:42:12.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.429+0000 7f5406d45700 1 Processor -- start 2023-07-14T17:42:12.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.429+0000 7f5406d45700 1 -- start start 2023-07-14T17:42:12.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.429+0000 7f5406d45700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54000fe280 0x7f5400193f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:12.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.429+0000 7f5406d45700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54000ff470 0x7f54001944b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:12.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.429+0000 7f5406d45700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54000ffe00 0x7f540019b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:12.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.429+0000 7f5406d45700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540019baf0 con 0x7f54000ff470 2023-07-14T17:42:12.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.429+0000 7f5406d45700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540019bc30 con 0x7f54000ffe00 2023-07-14T17:42:12.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.429+0000 7f5406d45700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540019bdc0 con 0x7f54000fe280 2023-07-14T17:42:12.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.429+0000 7f53fffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54000ff470 0x7f54001944b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:12.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.429+0000 7f53fffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54000ff470 0x7f54001944b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41130/0 (socket says 172.21.15.88:41130) 2023-07-14T17:42:12.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.429+0000 7f53fffff700 1 -- 172.21.15.88:0/3013174925 learned_addr learned my addr 172.21.15.88:0/3013174925 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:12.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.429+0000 7f5404ae1700 1 --2- 172.21.15.88:0/3013174925 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54000fe280 0x7f5400193f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:12.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.429+0000 7f54052e2700 1 --2- 172.21.15.88:0/3013174925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54000ffe00 0x7f540019b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:12.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.430+0000 7f53fffff700 1 -- 172.21.15.88:0/3013174925 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54000fe280 msgr2=0x7f5400193f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:12.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.430+0000 7f53fffff700 1 --2- 172.21.15.88:0/3013174925 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54000fe280 0x7f5400193f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.430+0000 7f53fffff700 1 -- 172.21.15.88:0/3013174925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54000ffe00 msgr2=0x7f540019b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:12.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.430+0000 7f53fffff700 1 --2- 172.21.15.88:0/3013174925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54000ffe00 0x7f540019b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.430+0000 7f53fffff700 1 -- 172.21.15.88:0/3013174925 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f8009d00 con 0x7f54000ff470 2023-07-14T17:42:12.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.430+0000 7f54052e2700 1 --2- 172.21.15.88:0/3013174925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54000ffe00 0x7f540019b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:42:12.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.430+0000 7f53fffff700 1 --2- 172.21.15.88:0/3013174925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54000ff470 0x7f54001944b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00e960 tx=0x7f53ec00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:12.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.430+0000 7f53fdffb700 1 -- 172.21.15.88:0/3013174925 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec00cb70 con 0x7f54000ff470 2023-07-14T17:42:12.419 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.430+0000 7f53fdffb700 1 -- 172.21.15.88:0/3013174925 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f53ec00eeb0 con 0x7f54000ff470 2023-07-14T17:42:12.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.430+0000 7f5406d45700 1 -- 172.21.15.88:0/3013174925 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5400066f70 con 0x7f54000ff470 2023-07-14T17:42:12.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.430+0000 7f53fdffb700 1 -- 172.21.15.88:0/3013174925 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53ec01a660 con 0x7f54000ff470 2023-07-14T17:42:12.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.430+0000 7f5406d45700 1 -- 172.21.15.88:0/3013174925 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54000674d0 con 0x7f54000ff470 2023-07-14T17:42:12.420 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.431+0000 7f5406d45700 1 -- 172.21.15.88:0/3013174925 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5400109920 con 0x7f54000ff470 2023-07-14T17:42:12.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.432+0000 7f53fdffb700 1 -- 172.21.15.88:0/3013174925 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f53ec011470 con 0x7f54000ff470 2023-07-14T17:42:12.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.432+0000 7f53fdffb700 1 --2- 172.21.15.88:0/3013174925 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f53f00a1220 0x7f53f00a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:12.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.432+0000 7f5404ae1700 1 --2- 172.21.15.88:0/3013174925 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f53f00a1220 0x7f53f00a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:12.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.433+0000 7f53fdffb700 1 -- 172.21.15.88:0/3013174925 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f53ec0cd560 con 0x7f54000ff470 2023-07-14T17:42:12.421 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.433+0000 7f5404ae1700 1 --2- 172.21.15.88:0/3013174925 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f53f00a1220 0x7f53f00a36d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f54000ff2d0 tx=0x7f53f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:12.422 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.437+0000 7f53fdffb700 1 -- 172.21.15.88:0/3013174925 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f53ec097150 con 0x7f54000ff470 2023-07-14T17:42:12.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.667+0000 7f5406d45700 1 -- 172.21.15.88:0/3013174925 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5400105130 con 0x7f54000ff470 2023-07-14T17:42:12.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.668+0000 7f53fdffb700 1 -- 172.21.15.88:0/3013174925 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f53ec09a770 con 0x7f54000ff470 2023-07-14T17:42:12.650 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:12.651 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-07-14T17:42:12.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.669+0000 7f5406d45700 1 -- 172.21.15.88:0/3013174925 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f53f00a1220 msgr2=0x7f53f00a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:12.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.669+0000 7f5406d45700 1 --2- 172.21.15.88:0/3013174925 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f53f00a1220 0x7f53f00a36d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f54000ff2d0 tx=0x7f53f4009260 comp rx=0 tx=0).stop 2023-07-14T17:42:12.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.670+0000 7f5406d45700 1 -- 172.21.15.88:0/3013174925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54000ff470 msgr2=0x7f54001944b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:12.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.670+0000 7f5406d45700 1 --2- 172.21.15.88:0/3013174925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54000ff470 0x7f54001944b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f53ec00e960 tx=0x7f53ec00ec70 comp rx=0 tx=0).stop 2023-07-14T17:42:12.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.670+0000 7f5406d45700 1 -- 172.21.15.88:0/3013174925 shutdown_connections 2023-07-14T17:42:12.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.670+0000 7f5406d45700 1 --2- 172.21.15.88:0/3013174925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f54000ffe00 0x7f540019b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.670+0000 7f5406d45700 1 --2- 172.21.15.88:0/3013174925 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f54000fe280 0x7f5400193f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.670+0000 7f5406d45700 1 --2- 172.21.15.88:0/3013174925 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f53f00a1220 0x7f53f00a36d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.670+0000 7f5406d45700 1 --2- 172.21.15.88:0/3013174925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54000ff470 0x7f54001944b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:12.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.670+0000 7f5406d45700 1 -- 172.21.15.88:0/3013174925 >> 172.21.15.88:0/3013174925 conn(0x7f54000f98a0 msgr2=0x7f54000fbad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:12.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.670+0000 7f5406d45700 1 -- 172.21.15.88:0/3013174925 shutdown_connections 2023-07-14T17:42:12.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:12.670+0000 7f5406d45700 1 -- 172.21.15.88:0/3013174925 wait complete. 2023-07-14T17:42:12.654 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 1 2023-07-14T17:42:12.849 INFO:journalctl@ceph.mds.l.smithi178.stdout:Jul 14 17:42:12 smithi178 systemd[1]: Starting Ceph mds.l for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:42:13.112 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:12 smithi178 ceph-mon[129368]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:13.113 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:12 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2907831292' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:13.113 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:12 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3379038062' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:42:13.113 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:12 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3013174925' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:13.113 INFO:journalctl@ceph.mds.l.smithi178.stdout:Jul 14 17:42:12 smithi178 podman[154530]: 2023-07-14T17:42:13.113 INFO:journalctl@ceph.mds.l.smithi178.stdout:Jul 14 17:42:13 smithi178 ceph-mds[154570]: -- 172.21.15.178:0/1936485536 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a3e3f991e0 con 0x55a3e3f6ec00 2023-07-14T17:42:13.114 INFO:journalctl@ceph.mds.l.smithi178.stdout:Jul 14 17:42:13 smithi178 bash[154530]: e3c2b8f1de44e721b8b8344cb6dccef3ca09f8a065868416223d6d3d22d16f8e 2023-07-14T17:42:13.114 INFO:journalctl@ceph.mds.l.smithi178.stdout:Jul 14 17:42:13 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-l[154546]: starting mds.l at 2023-07-14T17:42:13.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:12 smithi088 ceph-mon[127465]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:13.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:12 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2907831292' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:13.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:12 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3379038062' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:42:13.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:12 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3013174925' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:13.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:12 smithi107 ceph-mon[128999]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:13.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:12 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2907831292' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:13.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:12 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3379038062' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:42:13.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:12 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3013174925' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:13.414 INFO:journalctl@ceph.mds.l.smithi178.stdout:Jul 14 17:42:13 smithi178 systemd[1]: Started Ceph mds.l for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:42:14.385 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:14 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.178:6832/1716372869,v1:172.21.15.178:6833/1716372869] up:boot 2023-07-14T17:42:14.385 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:14 smithi107 ceph-mon[128999]: fsmap 1 up:standby 2023-07-14T17:42:14.385 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T17:42:14.386 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:14.386 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:14.386 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:14.386 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:14.386 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:14.386 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:14 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:14.386 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:14 smithi107 ceph-mon[128999]: Deploying daemon mds.k on smithi107 2023-07-14T17:42:14.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:14 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.178:6832/1716372869,v1:172.21.15.178:6833/1716372869] up:boot 2023-07-14T17:42:14.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:14 smithi178 ceph-mon[129368]: fsmap 1 up:standby 2023-07-14T17:42:14.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T17:42:14.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:14.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:14.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:14.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:14.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:14.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:14 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:14.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:14 smithi178 ceph-mon[129368]: Deploying daemon mds.k on smithi107 2023-07-14T17:42:14.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:14 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.178:6832/1716372869,v1:172.21.15.178:6833/1716372869] up:boot 2023-07-14T17:42:14.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:14 smithi088 ceph-mon[127465]: fsmap 1 up:standby 2023-07-14T17:42:14.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T17:42:14.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:14.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:14.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:14.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:14.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:14.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:14 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:14.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:14 smithi088 ceph-mon[127465]: Deploying daemon mds.k on smithi107 2023-07-14T17:42:14.665 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:42:14.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.817+0000 7f06ab831700 1 -- 172.21.15.88:0/70866751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06a40ffd10 msgr2=0x7f06a4100160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:14.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.817+0000 7f06ab831700 1 --2- 172.21.15.88:0/70866751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06a40ffd10 0x7f06a4100160 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009990 tx=0x7f06a0009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:14.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.817+0000 7f06ab831700 1 -- 172.21.15.88:0/70866751 shutdown_connections 2023-07-14T17:42:14.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.817+0000 7f06ab831700 1 --2- 172.21.15.88:0/70866751 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06a41006a0 0x7f06a4106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:14.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.817+0000 7f06ab831700 1 --2- 172.21.15.88:0/70866751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06a40feb20 0x7f06a40fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:14.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.817+0000 7f06ab831700 1 --2- 172.21.15.88:0/70866751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06a40ffd10 0x7f06a4100160 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:14.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.817+0000 7f06ab831700 1 -- 172.21.15.88:0/70866751 >> 172.21.15.88:0/70866751 conn(0x7f06a40fa100 msgr2=0x7f06a40fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:14.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.817+0000 7f06ab831700 1 -- 172.21.15.88:0/70866751 shutdown_connections 2023-07-14T17:42:14.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.817+0000 7f06ab831700 1 -- 172.21.15.88:0/70866751 wait complete. 2023-07-14T17:42:14.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06ab831700 1 Processor -- start 2023-07-14T17:42:14.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06ab831700 1 -- start start 2023-07-14T17:42:14.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06ab831700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06a40feb20 0x7f06a4069360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:14.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06ab831700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06a40ffd10 0x7f06a419a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:14.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06ab831700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06a41006a0 0x7f06a419cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:14.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06ab831700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a419d370 con 0x7f06a40feb20 2023-07-14T17:42:14.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06ab831700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a419d4e0 con 0x7f06a41006a0 2023-07-14T17:42:14.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06ab831700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a419d6a0 con 0x7f06a40ffd10 2023-07-14T17:42:14.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06a95cd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06a40feb20 0x7f06a4069360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:14.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06a95cd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06a40feb20 0x7f06a4069360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41142/0 (socket says 172.21.15.88:41142) 2023-07-14T17:42:14.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06a95cd700 1 -- 172.21.15.88:0/1002155189 learned_addr learned my addr 172.21.15.88:0/1002155189 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:14.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06a9dce700 1 --2- 172.21.15.88:0/1002155189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06a41006a0 0x7f06a419cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:14.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06a8dcc700 1 --2- 172.21.15.88:0/1002155189 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06a40ffd10 0x7f06a419a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:14.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06a95cd700 1 -- 172.21.15.88:0/1002155189 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06a40ffd10 msgr2=0x7f06a419a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:14.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06a95cd700 1 --2- 172.21.15.88:0/1002155189 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06a40ffd10 0x7f06a419a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:14.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06a95cd700 1 -- 172.21.15.88:0/1002155189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06a41006a0 msgr2=0x7f06a419cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:14.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06a95cd700 1 --2- 172.21.15.88:0/1002155189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06a41006a0 0x7f06a419cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:14.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06a95cd700 1 -- 172.21.15.88:0/1002155189 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a00095f0 con 0x7f06a40feb20 2023-07-14T17:42:14.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.818+0000 7f06a9dce700 1 --2- 172.21.15.88:0/1002155189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06a41006a0 0x7f06a419cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:42:14.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.819+0000 7f06a95cd700 1 --2- 172.21.15.88:0/1002155189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06a40feb20 0x7f06a4069360 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f069400c970 tx=0x7f069400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:14.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.819+0000 7f069a7fc700 1 -- 172.21.15.88:0/1002155189 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694004220 con 0x7f06a40feb20 2023-07-14T17:42:14.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.819+0000 7f069a7fc700 1 -- 172.21.15.88:0/1002155189 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0694007d30 con 0x7f06a40feb20 2023-07-14T17:42:14.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.819+0000 7f06ab831700 1 -- 172.21.15.88:0/1002155189 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a419d980 con 0x7f06a40feb20 2023-07-14T17:42:14.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.819+0000 7f069a7fc700 1 -- 172.21.15.88:0/1002155189 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694003bf0 con 0x7f06a40feb20 2023-07-14T17:42:14.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.819+0000 7f06ab831700 1 -- 172.21.15.88:0/1002155189 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a419de90 con 0x7f06a40feb20 2023-07-14T17:42:14.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.820+0000 7f06ab831700 1 -- 172.21.15.88:0/1002155189 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a4105a50 con 0x7f06a40feb20 2023-07-14T17:42:14.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.821+0000 7f069a7fc700 1 -- 172.21.15.88:0/1002155189 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f06940078a0 con 0x7f06a40feb20 2023-07-14T17:42:14.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.821+0000 7f069a7fc700 1 --2- 172.21.15.88:0/1002155189 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f06900a11a0 0x7f06900a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:14.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.822+0000 7f06a8dcc700 1 --2- 172.21.15.88:0/1002155189 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f06900a11a0 0x7f06900a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:14.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.822+0000 7f069a7fc700 1 -- 172.21.15.88:0/1002155189 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f06940cc0f0 con 0x7f06a40feb20 2023-07-14T17:42:14.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.822+0000 7f06a8dcc700 1 --2- 172.21.15.88:0/1002155189 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f06900a11a0 0x7f06900a3650 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f06a000c010 tx=0x7f06a000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:14.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:14.826+0000 7f069a7fc700 1 -- 172.21.15.88:0/1002155189 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0694095ce0 con 0x7f06a40feb20 2023-07-14T17:42:14.948 INFO:journalctl@ceph.mds.k.smithi107.stdout:Jul 14 17:42:14 smithi107 systemd[1]: Starting Ceph mds.k for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:42:15.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:15.060+0000 7f06ab831700 1 -- 172.21.15.88:0/1002155189 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06a402c8e0 con 0x7f06a40feb20 2023-07-14T17:42:15.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:15.060+0000 7f069a7fc700 1 -- 172.21.15.88:0/1002155189 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7f0694099300 con 0x7f06a40feb20 2023-07-14T17:42:15.043 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:15.043 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-14T17:42:15.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:15.062+0000 7f06ab831700 1 -- 172.21.15.88:0/1002155189 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f06900a11a0 msgr2=0x7f06900a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:15.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:15.062+0000 7f06ab831700 1 --2- 172.21.15.88:0/1002155189 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f06900a11a0 0x7f06900a3650 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f06a000c010 tx=0x7f06a000b540 comp rx=0 tx=0).stop 2023-07-14T17:42:15.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:15.062+0000 7f06ab831700 1 -- 172.21.15.88:0/1002155189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06a40feb20 msgr2=0x7f06a4069360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:15.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:15.062+0000 7f06ab831700 1 --2- 172.21.15.88:0/1002155189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06a40feb20 0x7f06a4069360 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f069400c970 tx=0x7f069400cd30 comp rx=0 tx=0).stop 2023-07-14T17:42:15.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:15.062+0000 7f06ab831700 1 -- 172.21.15.88:0/1002155189 shutdown_connections 2023-07-14T17:42:15.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:15.062+0000 7f06ab831700 1 --2- 172.21.15.88:0/1002155189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f06a41006a0 0x7f06a419cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:15.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:15.063+0000 7f06ab831700 1 --2- 172.21.15.88:0/1002155189 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f06900a11a0 0x7f06900a3650 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:15.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:15.063+0000 7f06ab831700 1 --2- 172.21.15.88:0/1002155189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06a40feb20 0x7f06a4069360 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:15.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:15.063+0000 7f06ab831700 1 --2- 172.21.15.88:0/1002155189 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f06a40ffd10 0x7f06a419a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:15.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:15.063+0000 7f06ab831700 1 -- 172.21.15.88:0/1002155189 >> 172.21.15.88:0/1002155189 conn(0x7f06a40fa100 msgr2=0x7f06a40fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:15.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:15.063+0000 7f06ab831700 1 -- 172.21.15.88:0/1002155189 shutdown_connections 2023-07-14T17:42:15.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:15.063+0000 7f06ab831700 1 -- 172.21.15.88:0/1002155189 wait complete. 2023-07-14T17:42:15.047 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 2 2023-07-14T17:42:15.202 INFO:journalctl@ceph.mds.k.smithi107.stdout:Jul 14 17:42:14 smithi107 podman[153047]: 2023-07-14T17:42:15.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:15 smithi088 ceph-mon[127465]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:15.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:15 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1002155189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:15.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:15 smithi107 ceph-mon[128999]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:15.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:15 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1002155189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:15.475 INFO:journalctl@ceph.mds.k.smithi107.stdout:Jul 14 17:42:15 smithi107 ceph-mds[153087]: -- 172.21.15.107:0/3167524278 <== mon.2 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d2a88511e0 con 0x55d2a8826800 2023-07-14T17:42:15.475 INFO:journalctl@ceph.mds.k.smithi107.stdout:Jul 14 17:42:15 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-k[153063]: starting mds.k at 2023-07-14T17:42:15.476 INFO:journalctl@ceph.mds.k.smithi107.stdout:Jul 14 17:42:15 smithi107 bash[153047]: 8f79948b4e4dbab4e639c421de8ef65c63781f7a65bca04fc4e56c9c4030c98b 2023-07-14T17:42:15.476 INFO:journalctl@ceph.mds.k.smithi107.stdout:Jul 14 17:42:15 smithi107 systemd[1]: Started Ceph mds.k for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:42:15.664 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:15 smithi178 ceph-mon[129368]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:15.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:15 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1002155189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:16.551 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:16 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:16.551 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:16 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:16.551 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:16 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:16.551 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:16 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:16.551 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:16 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:16.552 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:16 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:16.552 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:16 smithi088 ceph-mon[127465]: Deploying daemon mds.j on smithi088 2023-07-14T17:42:16.552 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:16 smithi088 ceph-mon[127465]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:16.552 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:16 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.107:6832/2946270337,v1:172.21.15.107:6833/2946270337] up:boot 2023-07-14T17:42:16.552 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:16 smithi088 ceph-mon[127465]: fsmap 2 up:standby 2023-07-14T17:42:16.552 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:16 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T17:42:16.664 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:16 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:16.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:16 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:16.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:16 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:16.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:16 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:16.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:16 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:16.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:16 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:16.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:16 smithi178 ceph-mon[129368]: Deploying daemon mds.j on smithi088 2023-07-14T17:42:16.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:16 smithi178 ceph-mon[129368]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:16.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:16 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.107:6832/2946270337,v1:172.21.15.107:6833/2946270337] up:boot 2023-07-14T17:42:16.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:16 smithi178 ceph-mon[129368]: fsmap 2 up:standby 2023-07-14T17:42:16.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:16 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T17:42:16.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:16 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:16.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:16 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:16.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:16 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:16.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:16 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:16.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:16 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:16.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:16 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:16.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:16 smithi107 ceph-mon[128999]: Deploying daemon mds.j on smithi088 2023-07-14T17:42:16.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:16 smithi107 ceph-mon[128999]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:16.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:16 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.107:6832/2946270337,v1:172.21.15.107:6833/2946270337] up:boot 2023-07-14T17:42:16.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:16 smithi107 ceph-mon[128999]: fsmap 2 up:standby 2023-07-14T17:42:16.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:16 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T17:42:16.942 INFO:journalctl@ceph.mds.j.smithi088.stdout:Jul 14 17:42:16 smithi088 systemd[1]: Starting Ceph mds.j for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:42:17.057 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:42:17.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.206+0000 7f7cbf296700 1 -- 172.21.15.88:0/2173089792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb81017b0 msgr2=0x7f7cb8108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:17.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.206+0000 7f7cbf296700 1 --2- 172.21.15.88:0/2173089792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb81017b0 0x7f7cb8108070 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7cb000b0a0 tx=0x7f7cb000b3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:17.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.206+0000 7f7cbf296700 1 -- 172.21.15.88:0/2173089792 shutdown_connections 2023-07-14T17:42:17.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.206+0000 7f7cbf296700 1 --2- 172.21.15.88:0/2173089792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb81017b0 0x7f7cb8108070 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:17.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.206+0000 7f7cbf296700 1 --2- 172.21.15.88:0/2173089792 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cb80ffc30 0x7f7cb8100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:17.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.206+0000 7f7cbf296700 1 --2- 172.21.15.88:0/2173089792 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7cb8100e20 0x7f7cb8101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:17.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.206+0000 7f7cbf296700 1 -- 172.21.15.88:0/2173089792 >> 172.21.15.88:0/2173089792 conn(0x7f7cb80fb210 msgr2=0x7f7cb80fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:17.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.207+0000 7f7cbf296700 1 -- 172.21.15.88:0/2173089792 shutdown_connections 2023-07-14T17:42:17.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.207+0000 7f7cbf296700 1 -- 172.21.15.88:0/2173089792 wait complete. 2023-07-14T17:42:17.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.207+0000 7f7cbf296700 1 Processor -- start 2023-07-14T17:42:17.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.207+0000 7f7cbf296700 1 -- start start 2023-07-14T17:42:17.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.207+0000 7f7cbf296700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7cb80ffc30 0x7f7cb81958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:17.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.207+0000 7f7cbf296700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cb8100e20 0x7f7cb8195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:17.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.207+0000 7f7cbf296700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb81017b0 0x7f7cb819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:17.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.207+0000 7f7cbf296700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb819d450 con 0x7f7cb81017b0 2023-07-14T17:42:17.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.207+0000 7f7cbf296700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb819d590 con 0x7f7cb8100e20 2023-07-14T17:42:17.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.207+0000 7f7cbf296700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb819d720 con 0x7f7cb80ffc30 2023-07-14T17:42:17.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.207+0000 7f7cbd833700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb81017b0 0x7f7cb819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:17.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.207+0000 7f7cbd833700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb81017b0 0x7f7cb819ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41158/0 (socket says 172.21.15.88:41158) 2023-07-14T17:42:17.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.207+0000 7f7cbd833700 1 -- 172.21.15.88:0/2291130890 learned_addr learned my addr 172.21.15.88:0/2291130890 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:17.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.207+0000 7f7cbc831700 1 --2- 172.21.15.88:0/2291130890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cb8100e20 0x7f7cb8195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:17.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.208+0000 7f7cbd032700 1 --2- 172.21.15.88:0/2291130890 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7cb80ffc30 0x7f7cb81958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:17.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.208+0000 7f7cbd833700 1 -- 172.21.15.88:0/2291130890 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7cb80ffc30 msgr2=0x7f7cb81958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:17.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.208+0000 7f7cbd833700 1 --2- 172.21.15.88:0/2291130890 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7cb80ffc30 0x7f7cb81958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:17.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.208+0000 7f7cbd833700 1 -- 172.21.15.88:0/2291130890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cb8100e20 msgr2=0x7f7cb8195e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:17.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.208+0000 7f7cbd833700 1 --2- 172.21.15.88:0/2291130890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cb8100e20 0x7f7cb8195e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:17.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.208+0000 7f7cbd833700 1 -- 172.21.15.88:0/2291130890 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb0009d00 con 0x7f7cb81017b0 2023-07-14T17:42:17.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.208+0000 7f7cbd032700 1 --2- 172.21.15.88:0/2291130890 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7cb80ffc30 0x7f7cb81958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:42:17.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.208+0000 7f7cbd833700 1 --2- 172.21.15.88:0/2291130890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb81017b0 0x7f7cb819ce80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7cb00095e0 tx=0x7f7cb000ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:17.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.208+0000 7f7cae7fc700 1 -- 172.21.15.88:0/2291130890 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb000e040 con 0x7f7cb81017b0 2023-07-14T17:42:17.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.208+0000 7f7cbf296700 1 -- 172.21.15.88:0/2291130890 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb819d9a0 con 0x7f7cb81017b0 2023-07-14T17:42:17.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.208+0000 7f7cae7fc700 1 -- 172.21.15.88:0/2291130890 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7cb0003c00 con 0x7f7cb81017b0 2023-07-14T17:42:17.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.208+0000 7f7cae7fc700 1 -- 172.21.15.88:0/2291130890 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb00052a0 con 0x7f7cb81017b0 2023-07-14T17:42:17.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.208+0000 7f7cbf296700 1 -- 172.21.15.88:0/2291130890 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb819de20 con 0x7f7cb81017b0 2023-07-14T17:42:17.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.209+0000 7f7cbf296700 1 -- 172.21.15.88:0/2291130890 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb8107130 con 0x7f7cb81017b0 2023-07-14T17:42:17.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.210+0000 7f7cae7fc700 1 -- 172.21.15.88:0/2291130890 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7cb001d470 con 0x7f7cb81017b0 2023-07-14T17:42:17.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.210+0000 7f7cae7fc700 1 --2- 172.21.15.88:0/2291130890 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7ca40a1220 0x7f7ca40a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:17.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.211+0000 7f7cbd032700 1 --2- 172.21.15.88:0/2291130890 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7ca40a1220 0x7f7ca40a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:17.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.211+0000 7f7cae7fc700 1 -- 172.21.15.88:0/2291130890 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7cb00cd7b0 con 0x7f7cb81017b0 2023-07-14T17:42:17.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.211+0000 7f7cbd032700 1 --2- 172.21.15.88:0/2291130890 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7ca40a1220 0x7f7ca40a36d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8100c80 tx=0x7f7ca8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:17.197 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.215+0000 7f7cae7fc700 1 -- 172.21.15.88:0/2291130890 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7cb0097390 con 0x7f7cb81017b0 2023-07-14T17:42:17.347 INFO:journalctl@ceph.mds.j.smithi088.stdout:Jul 14 17:42:17 smithi088 podman[160716]: 2023-07-14T17:42:17.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.475+0000 7f7cbf296700 1 -- 172.21.15.88:0/2291130890 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cb802cd70 con 0x7f7cb81017b0 2023-07-14T17:42:17.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.475+0000 7f7cae7fc700 1 -- 172.21.15.88:0/2291130890 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2020 (secure 0 0 0) 0x7f7cb009a9b0 con 0x7f7cb81017b0 2023-07-14T17:42:17.458 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:17.458 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-14T17:42:17.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.477+0000 7f7cbf296700 1 -- 172.21.15.88:0/2291130890 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7ca40a1220 msgr2=0x7f7ca40a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:17.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.477+0000 7f7cbf296700 1 --2- 172.21.15.88:0/2291130890 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7ca40a1220 0x7f7ca40a36d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8100c80 tx=0x7f7ca8009260 comp rx=0 tx=0).stop 2023-07-14T17:42:17.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.477+0000 7f7cbf296700 1 -- 172.21.15.88:0/2291130890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb81017b0 msgr2=0x7f7cb819ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:17.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.477+0000 7f7cbf296700 1 --2- 172.21.15.88:0/2291130890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb81017b0 0x7f7cb819ce80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7cb00095e0 tx=0x7f7cb000ba50 comp rx=0 tx=0).stop 2023-07-14T17:42:17.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.477+0000 7f7cbf296700 1 -- 172.21.15.88:0/2291130890 shutdown_connections 2023-07-14T17:42:17.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.477+0000 7f7cbf296700 1 --2- 172.21.15.88:0/2291130890 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7ca40a1220 0x7f7ca40a36d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:17.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.477+0000 7f7cbf296700 1 --2- 172.21.15.88:0/2291130890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cb81017b0 0x7f7cb819ce80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:17.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.477+0000 7f7cbf296700 1 --2- 172.21.15.88:0/2291130890 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7cb80ffc30 0x7f7cb81958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:17.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.477+0000 7f7cbf296700 1 --2- 172.21.15.88:0/2291130890 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7cb8100e20 0x7f7cb8195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:17.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.477+0000 7f7cbf296700 1 -- 172.21.15.88:0/2291130890 >> 172.21.15.88:0/2291130890 conn(0x7f7cb80fb210 msgr2=0x7f7cb8104270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:17.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.477+0000 7f7cbf296700 1 -- 172.21.15.88:0/2291130890 shutdown_connections 2023-07-14T17:42:17.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:17.477+0000 7f7cbf296700 1 -- 172.21.15.88:0/2291130890 wait complete. 2023-07-14T17:42:17.462 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 3 2023-07-14T17:42:17.692 INFO:journalctl@ceph.mds.j.smithi088.stdout:Jul 14 17:42:17 smithi088 ceph-mds[160799]: -- 172.21.15.88:0/2381197617 <== mon.0 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e23ac9f1e0 con 0x55e23ac74c00 2023-07-14T17:42:17.693 INFO:journalctl@ceph.mds.j.smithi088.stdout:Jul 14 17:42:17 smithi088 bash[160716]: b69aea97d5e597a39f0cf3e8a3fee51e2e64b6b4c425bf4a9afe76dd9c7fb105 2023-07-14T17:42:17.693 INFO:journalctl@ceph.mds.j.smithi088.stdout:Jul 14 17:42:17 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-j[160755]: starting mds.j at 2023-07-14T17:42:17.693 INFO:journalctl@ceph.mds.j.smithi088.stdout:Jul 14 17:42:17 smithi088 systemd[1]: Started Ceph mds.j for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:42:18.664 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:18.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:18.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:18.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:18.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:18.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:18.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:18 smithi178 ceph-mon[129368]: Deploying daemon mds.d on smithi088 2023-07-14T17:42:18.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:18 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2291130890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:18.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:18 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.88:6834/2870142745,v1:172.21.15.88:6835/2870142745] up:boot 2023-07-14T17:42:18.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:18 smithi178 ceph-mon[129368]: fsmap 3 up:standby 2023-07-14T17:42:18.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:18 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T17:42:18.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:18 smithi178 ceph-mon[129368]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:18.692 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:18.693 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:18.693 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:18.693 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:18.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:18.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:18.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:18 smithi088 ceph-mon[127465]: Deploying daemon mds.d on smithi088 2023-07-14T17:42:18.694 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:18 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2291130890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:18.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:18 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.88:6834/2870142745,v1:172.21.15.88:6835/2870142745] up:boot 2023-07-14T17:42:18.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:18 smithi088 ceph-mon[127465]: fsmap 3 up:standby 2023-07-14T17:42:18.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:18 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T17:42:18.695 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:18 smithi088 ceph-mon[127465]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:18.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:18.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:18.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:18.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:18.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:18.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:18.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:18 smithi107 ceph-mon[128999]: Deploying daemon mds.d on smithi088 2023-07-14T17:42:18.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:18 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2291130890' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:18.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:18 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.88:6834/2870142745,v1:172.21.15.88:6835/2870142745] up:boot 2023-07-14T17:42:18.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:18 smithi107 ceph-mon[128999]: fsmap 3 up:standby 2023-07-14T17:42:18.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:18 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T17:42:18.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:18 smithi107 ceph-mon[128999]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:19.192 INFO:journalctl@ceph.mds.d.smithi088.stdout:Jul 14 17:42:18 smithi088 systemd[1]: Starting Ceph mds.d for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:42:19.473 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:42:19.501 INFO:journalctl@ceph.mds.d.smithi088.stdout:Jul 14 17:42:19 smithi088 podman[161101]: 2023-07-14T17:42:19.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.624+0000 7fce92e17700 1 -- 172.21.15.88:0/1283968453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce8c1006e0 msgr2=0x7fce8c106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:19.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.624+0000 7fce92e17700 1 --2- 172.21.15.88:0/1283968453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce8c1006e0 0x7fce8c106f80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fce8400b0a0 tx=0x7fce8400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:19.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.624+0000 7fce92e17700 1 -- 172.21.15.88:0/1283968453 shutdown_connections 2023-07-14T17:42:19.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.624+0000 7fce92e17700 1 --2- 172.21.15.88:0/1283968453 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce8c1006e0 0x7fce8c106f80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:19.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.624+0000 7fce92e17700 1 --2- 172.21.15.88:0/1283968453 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce8c0ff020 0x7fce8c0ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:19.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.624+0000 7fce92e17700 1 --2- 172.21.15.88:0/1283968453 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce8c0ffd30 0x7fce8c1001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:19.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.624+0000 7fce92e17700 1 -- 172.21.15.88:0/1283968453 >> 172.21.15.88:0/1283968453 conn(0x7fce8c0f98a0 msgr2=0x7fce8c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:19.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.624+0000 7fce92e17700 1 -- 172.21.15.88:0/1283968453 shutdown_connections 2023-07-14T17:42:19.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.624+0000 7fce92e17700 1 -- 172.21.15.88:0/1283968453 wait complete. 2023-07-14T17:42:19.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.624+0000 7fce92e17700 1 Processor -- start 2023-07-14T17:42:19.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce92e17700 1 -- start start 2023-07-14T17:42:19.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce92e17700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce8c0ff020 0x7fce8c195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:19.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce92e17700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce8c0ffd30 0x7fce8c195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:19.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce92e17700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce8c1006e0 0x7fce8c19cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:19.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce92e17700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c19d490 con 0x7fce8c0ff020 2023-07-14T17:42:19.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce92e17700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c19d5d0 con 0x7fce8c0ffd30 2023-07-14T17:42:19.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce92e17700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8c19d760 con 0x7fce8c1006e0 2023-07-14T17:42:19.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce90bb3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce8c0ff020 0x7fce8c195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:19.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce90bb3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce8c0ff020 0x7fce8c195910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41194/0 (socket says 172.21.15.88:41194) 2023-07-14T17:42:19.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce90bb3700 1 -- 172.21.15.88:0/1850525446 learned_addr learned my addr 172.21.15.88:0/1850525446 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:19.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce8bfff700 1 --2- 172.21.15.88:0/1850525446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce8c0ffd30 0x7fce8c195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:19.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce913b4700 1 --2- 172.21.15.88:0/1850525446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce8c1006e0 0x7fce8c19cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:19.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce90bb3700 1 -- 172.21.15.88:0/1850525446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce8c1006e0 msgr2=0x7fce8c19cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:19.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce90bb3700 1 --2- 172.21.15.88:0/1850525446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce8c1006e0 0x7fce8c19cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:19.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce90bb3700 1 -- 172.21.15.88:0/1850525446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce8c0ffd30 msgr2=0x7fce8c195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:19.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce90bb3700 1 --2- 172.21.15.88:0/1850525446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce8c0ffd30 0x7fce8c195e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:19.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce90bb3700 1 -- 172.21.15.88:0/1850525446 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce84009d00 con 0x7fce8c0ff020 2023-07-14T17:42:19.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce90bb3700 1 --2- 172.21.15.88:0/1850525446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce8c0ff020 0x7fce8c195910 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fce8000e970 tx=0x7fce8000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:19.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce89ffb700 1 -- 172.21.15.88:0/1850525446 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce80009790 con 0x7fce8c0ff020 2023-07-14T17:42:19.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce89ffb700 1 -- 172.21.15.88:0/1850525446 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fce8000eec0 con 0x7fce8c0ff020 2023-07-14T17:42:19.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce92e17700 1 -- 172.21.15.88:0/1850525446 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce8c19da40 con 0x7fce8c0ff020 2023-07-14T17:42:19.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.625+0000 7fce89ffb700 1 -- 172.21.15.88:0/1850525446 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce80003c10 con 0x7fce8c0ff020 2023-07-14T17:42:19.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.626+0000 7fce92e17700 1 -- 172.21.15.88:0/1850525446 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce8c19df50 con 0x7fce8c0ff020 2023-07-14T17:42:19.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.627+0000 7fce92e17700 1 -- 172.21.15.88:0/1850525446 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce8c106cc0 con 0x7fce8c0ff020 2023-07-14T17:42:19.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.628+0000 7fce89ffb700 1 -- 172.21.15.88:0/1850525446 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fce80003db0 con 0x7fce8c0ff020 2023-07-14T17:42:19.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.628+0000 7fce89ffb700 1 --2- 172.21.15.88:0/1850525446 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fce7c0a11a0 0x7fce7c0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:19.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.628+0000 7fce89ffb700 1 -- 172.21.15.88:0/1850525446 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fce80014070 con 0x7fce8c0ff020 2023-07-14T17:42:19.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.628+0000 7fce8bfff700 1 --2- 172.21.15.88:0/1850525446 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fce7c0a11a0 0x7fce7c0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:19.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.629+0000 7fce8bfff700 1 --2- 172.21.15.88:0/1850525446 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fce7c0a11a0 0x7fce7c0a3650 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fce8c0fed40 tx=0x7fce78007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:19.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.633+0000 7fce89ffb700 1 -- 172.21.15.88:0/1850525446 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fce800962b0 con 0x7fce8c0ff020 2023-07-14T17:42:19.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.870+0000 7fce92e17700 1 -- 172.21.15.88:0/1850525446 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce8c02c8e0 con 0x7fce8c0ff020 2023-07-14T17:42:19.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.870+0000 7fce89ffb700 1 -- 172.21.15.88:0/1850525446 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3531 (secure 0 0 0) 0x7fce800998d0 con 0x7fce8c0ff020 2023-07-14T17:42:19.853 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:19.854 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-14T17:42:19.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.872+0000 7fce92e17700 1 -- 172.21.15.88:0/1850525446 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fce7c0a11a0 msgr2=0x7fce7c0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:19.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.872+0000 7fce92e17700 1 --2- 172.21.15.88:0/1850525446 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fce7c0a11a0 0x7fce7c0a3650 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fce8c0fed40 tx=0x7fce78007400 comp rx=0 tx=0).stop 2023-07-14T17:42:19.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.872+0000 7fce92e17700 1 -- 172.21.15.88:0/1850525446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce8c0ff020 msgr2=0x7fce8c195910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:19.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.872+0000 7fce92e17700 1 --2- 172.21.15.88:0/1850525446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce8c0ff020 0x7fce8c195910 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fce8000e970 tx=0x7fce8000ec80 comp rx=0 tx=0).stop 2023-07-14T17:42:19.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.872+0000 7fce92e17700 1 -- 172.21.15.88:0/1850525446 shutdown_connections 2023-07-14T17:42:19.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.872+0000 7fce92e17700 1 --2- 172.21.15.88:0/1850525446 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce8c1006e0 0x7fce8c19cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:19.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.872+0000 7fce92e17700 1 --2- 172.21.15.88:0/1850525446 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fce7c0a11a0 0x7fce7c0a3650 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:19.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.872+0000 7fce92e17700 1 --2- 172.21.15.88:0/1850525446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce8c0ff020 0x7fce8c195910 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:19.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.872+0000 7fce92e17700 1 --2- 172.21.15.88:0/1850525446 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce8c0ffd30 0x7fce8c195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:19.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.872+0000 7fce92e17700 1 -- 172.21.15.88:0/1850525446 >> 172.21.15.88:0/1850525446 conn(0x7fce8c0f98a0 msgr2=0x7fce8c0fbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:19.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.872+0000 7fce92e17700 1 -- 172.21.15.88:0/1850525446 shutdown_connections 2023-07-14T17:42:19.857 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:19.872+0000 7fce92e17700 1 -- 172.21.15.88:0/1850525446 wait complete. 2023-07-14T17:42:19.857 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 5 2023-07-14T17:42:19.866 INFO:journalctl@ceph.mds.d.smithi088.stdout:Jul 14 17:42:19 smithi088 ceph-mds[161147]: -- 172.21.15.88:0/3258666417 <== mon.0 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ac3471d1e0 con 0x55ac346f2800 2023-07-14T17:42:19.866 INFO:journalctl@ceph.mds.d.smithi088.stdout:Jul 14 17:42:19 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-d[161117]: starting mds.d at 2023-07-14T17:42:19.866 INFO:journalctl@ceph.mds.d.smithi088.stdout:Jul 14 17:42:19 smithi088 bash[161101]: ea6ec2155e1210c3a598d69f18d50dd633bd833726fad147b29dcf623756629d 2023-07-14T17:42:19.866 INFO:journalctl@ceph.mds.d.smithi088.stdout:Jul 14 17:42:19 smithi088 systemd[1]: Started Ceph mds.d for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:42:20.874 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:20 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:20.875 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:20 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:20.875 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:20 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:20.875 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:20 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:20.876 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:20 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:20.876 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:20 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:20.876 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:20 smithi107 ceph-mon[128999]: Deploying daemon mds.e on smithi107 2023-07-14T17:42:20.876 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:20 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.88:6836/783713410,v1:172.21.15.88:6837/783713410] up:boot 2023-07-14T17:42:20.877 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:20 smithi107 ceph-mon[128999]: fsmap 4 up:standby 2023-07-14T17:42:20.877 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:20 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T17:42:20.877 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:20 smithi107 ceph-mon[128999]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:20.878 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:20 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1850525446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:20.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:20.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:20.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:20.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:20.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:20.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:20.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:20 smithi178 ceph-mon[129368]: Deploying daemon mds.e on smithi107 2023-07-14T17:42:20.917 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:20 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.88:6836/783713410,v1:172.21.15.88:6837/783713410] up:boot 2023-07-14T17:42:20.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:20 smithi178 ceph-mon[129368]: fsmap 4 up:standby 2023-07-14T17:42:20.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:20 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T17:42:20.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:20 smithi178 ceph-mon[129368]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:20.918 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:20 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1850525446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:20.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:20 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:20.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:20 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:20.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:20 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:20.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:20 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:20.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:20 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:20.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:20 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:20.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:20 smithi088 ceph-mon[127465]: Deploying daemon mds.e on smithi107 2023-07-14T17:42:20.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:20 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.88:6836/783713410,v1:172.21.15.88:6837/783713410] up:boot 2023-07-14T17:42:20.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:20 smithi088 ceph-mon[127465]: fsmap 4 up:standby 2023-07-14T17:42:20.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:20 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T17:42:20.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:20 smithi088 ceph-mon[127465]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:20.945 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:20 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1850525446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:21.474 INFO:journalctl@ceph.mds.e.smithi107.stdout:Jul 14 17:42:21 smithi107 systemd[1]: Starting Ceph mds.e for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:42:21.770 INFO:journalctl@ceph.mds.e.smithi107.stdout:Jul 14 17:42:21 smithi107 podman[153386]: 2023-07-14T17:42:21.771 INFO:journalctl@ceph.mds.e.smithi107.stdout:Jul 14 17:42:21 smithi107 ceph-mds[153426]: -- 172.21.15.107:0/2986393151 <== mon.2 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562b7ab971e0 con 0x562b7ab6d000 2023-07-14T17:42:21.771 INFO:journalctl@ceph.mds.e.smithi107.stdout:Jul 14 17:42:21 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-e[153402]: starting mds.e at 2023-07-14T17:42:21.771 INFO:journalctl@ceph.mds.e.smithi107.stdout:Jul 14 17:42:21 smithi107 bash[153386]: 0127cf452a767b768fb058a855678c641829542a3ce2133509774420fa30a757 2023-07-14T17:42:21.868 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:42:21.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.016+0000 7fd189a46700 1 -- 172.21.15.88:0/603133162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd184100160 msgr2=0x7fd184100570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:21.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.016+0000 7fd189a46700 1 --2- 172.21.15.88:0/603133162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd184100160 0x7fd184100570 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd174009990 tx=0x7fd174009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:21.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.016+0000 7fd189a46700 1 -- 172.21.15.88:0/603133162 shutdown_connections 2023-07-14T17:42:22.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.016+0000 7fd189a46700 1 --2- 172.21.15.88:0/603133162 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1840ff760 0x7fd184107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:22.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.016+0000 7fd189a46700 1 --2- 172.21.15.88:0/603133162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd184100160 0x7fd184100570 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:22.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.016+0000 7fd189a46700 1 --2- 172.21.15.88:0/603133162 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1840fedb0 0x7fd1840ff220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:22.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.016+0000 7fd189a46700 1 -- 172.21.15.88:0/603133162 >> 172.21.15.88:0/603133162 conn(0x7fd1840f98d0 msgr2=0x7fd1840fbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:22.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.017+0000 7fd189a46700 1 -- 172.21.15.88:0/603133162 shutdown_connections 2023-07-14T17:42:22.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.017+0000 7fd189a46700 1 -- 172.21.15.88:0/603133162 wait complete. 2023-07-14T17:42:22.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.017+0000 7fd189a46700 1 Processor -- start 2023-07-14T17:42:22.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.017+0000 7fd189a46700 1 -- start start 2023-07-14T17:42:22.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.017+0000 7fd189a46700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1840fedb0 0x7fd184195810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:22.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.017+0000 7fd189a46700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1840ff760 0x7fd184195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:22.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.017+0000 7fd189a46700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd184100160 0x7fd18419cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:22.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.017+0000 7fd189a46700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18419d390 con 0x7fd1840fedb0 2023-07-14T17:42:22.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.017+0000 7fd189a46700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18419d4d0 con 0x7fd184100160 2023-07-14T17:42:22.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.017+0000 7fd189a46700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd18419d660 con 0x7fd1840ff760 2023-07-14T17:42:22.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.017+0000 7fd182ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1840fedb0 0x7fd184195810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:22.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.018+0000 7fd182ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1840fedb0 0x7fd184195810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41210/0 (socket says 172.21.15.88:41210) 2023-07-14T17:42:22.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.018+0000 7fd182ffd700 1 -- 172.21.15.88:0/3569275297 learned_addr learned my addr 172.21.15.88:0/3569275297 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:22.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.018+0000 7fd1837fe700 1 --2- 172.21.15.88:0/3569275297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd184100160 0x7fd18419cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:22.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.018+0000 7fd1827fc700 1 --2- 172.21.15.88:0/3569275297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1840ff760 0x7fd184195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:22.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.018+0000 7fd182ffd700 1 -- 172.21.15.88:0/3569275297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1840ff760 msgr2=0x7fd184195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:22.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.018+0000 7fd182ffd700 1 --2- 172.21.15.88:0/3569275297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1840ff760 0x7fd184195d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:22.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.018+0000 7fd182ffd700 1 -- 172.21.15.88:0/3569275297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd184100160 msgr2=0x7fd18419cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:22.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.018+0000 7fd182ffd700 1 --2- 172.21.15.88:0/3569275297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd184100160 0x7fd18419cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:22.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.018+0000 7fd182ffd700 1 -- 172.21.15.88:0/3569275297 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1740095f0 con 0x7fd1840fedb0 2023-07-14T17:42:22.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.018+0000 7fd1827fc700 1 --2- 172.21.15.88:0/3569275297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1840ff760 0x7fd184195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:42:22.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.018+0000 7fd182ffd700 1 --2- 172.21.15.88:0/3569275297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1840fedb0 0x7fd184195810 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd17400c010 tx=0x7fd174004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:22.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.018+0000 7fd188a44700 1 -- 172.21.15.88:0/3569275297 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17401d070 con 0x7fd1840fedb0 2023-07-14T17:42:22.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.018+0000 7fd188a44700 1 -- 172.21.15.88:0/3569275297 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd1740054c0 con 0x7fd1840fedb0 2023-07-14T17:42:22.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.018+0000 7fd189a46700 1 -- 172.21.15.88:0/3569275297 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18419d8e0 con 0x7fd1840fedb0 2023-07-14T17:42:22.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.018+0000 7fd188a44700 1 -- 172.21.15.88:0/3569275297 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd174003ab0 con 0x7fd1840fedb0 2023-07-14T17:42:22.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.018+0000 7fd189a46700 1 -- 172.21.15.88:0/3569275297 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18419dd00 con 0x7fd1840fedb0 2023-07-14T17:42:22.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.019+0000 7fd16a7fc700 1 -- 172.21.15.88:0/3569275297 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1640052f0 con 0x7fd1840fedb0 2023-07-14T17:42:22.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.020+0000 7fd188a44700 1 -- 172.21.15.88:0/3569275297 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd1740047a0 con 0x7fd1840fedb0 2023-07-14T17:42:22.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.021+0000 7fd188a44700 1 --2- 172.21.15.88:0/3569275297 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd16c0a1220 0x7fd16c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:22.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.021+0000 7fd1827fc700 1 --2- 172.21.15.88:0/3569275297 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd16c0a1220 0x7fd16c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:22.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.021+0000 7fd188a44700 1 -- 172.21.15.88:0/3569275297 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd1740cd980 con 0x7fd1840fedb0 2023-07-14T17:42:22.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.021+0000 7fd1827fc700 1 --2- 172.21.15.88:0/3569275297 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd16c0a1220 0x7fd16c0a36d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd178005950 tx=0x7fd1780058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:22.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.024+0000 7fd188a44700 1 -- 172.21.15.88:0/3569275297 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd174097680 con 0x7fd1840fedb0 2023-07-14T17:42:22.224 INFO:journalctl@ceph.mds.e.smithi107.stdout:Jul 14 17:42:21 smithi107 systemd[1]: Started Ceph mds.e for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:42:22.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.254+0000 7fd16a7fc700 1 -- 172.21.15.88:0/3569275297 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd164005160 con 0x7fd1840fedb0 2023-07-14T17:42:22.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.255+0000 7fd188a44700 1 -- 172.21.15.88:0/3569275297 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4291 (secure 0 0 0) 0x7fd17409aca0 con 0x7fd1840fedb0 2023-07-14T17:42:22.237 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:22.237 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3062637672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3062637672},{"type":"v1","addr":"172.21.15.107:6835","nonce":3062637672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-07-14T17:42:22.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.257+0000 7fd16a7fc700 1 -- 172.21.15.88:0/3569275297 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd16c0a1220 msgr2=0x7fd16c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:22.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.257+0000 7fd16a7fc700 1 --2- 172.21.15.88:0/3569275297 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd16c0a1220 0x7fd16c0a36d0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd178005950 tx=0x7fd1780058e0 comp rx=0 tx=0).stop 2023-07-14T17:42:22.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.257+0000 7fd16a7fc700 1 -- 172.21.15.88:0/3569275297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1840fedb0 msgr2=0x7fd184195810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:22.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.257+0000 7fd16a7fc700 1 --2- 172.21.15.88:0/3569275297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1840fedb0 0x7fd184195810 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd17400c010 tx=0x7fd174004730 comp rx=0 tx=0).stop 2023-07-14T17:42:22.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.257+0000 7fd16a7fc700 1 -- 172.21.15.88:0/3569275297 shutdown_connections 2023-07-14T17:42:22.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.257+0000 7fd16a7fc700 1 --2- 172.21.15.88:0/3569275297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd184100160 0x7fd18419cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:22.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.257+0000 7fd16a7fc700 1 --2- 172.21.15.88:0/3569275297 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd16c0a1220 0x7fd16c0a36d0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:22.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.257+0000 7fd16a7fc700 1 --2- 172.21.15.88:0/3569275297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1840fedb0 0x7fd184195810 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:22.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.257+0000 7fd16a7fc700 1 --2- 172.21.15.88:0/3569275297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1840ff760 0x7fd184195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:22.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.257+0000 7fd16a7fc700 1 -- 172.21.15.88:0/3569275297 >> 172.21.15.88:0/3569275297 conn(0x7fd1840f98d0 msgr2=0x7fd1841032b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:22.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.257+0000 7fd16a7fc700 1 -- 172.21.15.88:0/3569275297 shutdown_connections 2023-07-14T17:42:22.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:22.257+0000 7fd16a7fc700 1 -- 172.21.15.88:0/3569275297 wait complete. 2023-07-14T17:42:22.241 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 6 2023-07-14T17:42:23.131 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:22 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] up:boot 2023-07-14T17:42:23.131 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:22 smithi178 ceph-mon[129368]: fsmap 5 up:standby 2023-07-14T17:42:23.131 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:22 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T17:42:23.132 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:22 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:23.132 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:22 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:23.132 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:22 smithi178 ceph-mon[129368]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:23.133 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:22 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:23.133 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:22 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:23.133 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:22 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:23.133 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:22 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:23.134 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:22 smithi178 ceph-mon[129368]: Deploying daemon mds.f on smithi178 2023-07-14T17:42:23.134 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:22 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3569275297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:23.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:22 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] up:boot 2023-07-14T17:42:23.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:22 smithi088 ceph-mon[127465]: fsmap 5 up:standby 2023-07-14T17:42:23.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:22 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T17:42:23.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:22 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:23.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:22 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:23.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:22 smithi088 ceph-mon[127465]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:23.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:22 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:23.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:22 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:23.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:22 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:23.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:22 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:23.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:22 smithi088 ceph-mon[127465]: Deploying daemon mds.f on smithi178 2023-07-14T17:42:23.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:22 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3569275297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:23.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:22 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] up:boot 2023-07-14T17:42:23.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:22 smithi107 ceph-mon[128999]: fsmap 5 up:standby 2023-07-14T17:42:23.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:22 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T17:42:23.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:22 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:23.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:22 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:23.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:22 smithi107 ceph-mon[128999]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:23.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:22 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:23.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:22 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:23.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:22 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:23.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:22 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:23.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:22 smithi107 ceph-mon[128999]: Deploying daemon mds.f on smithi178 2023-07-14T17:42:23.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:22 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3569275297' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:23.382 INFO:journalctl@ceph.mds.f.smithi178.stdout:Jul 14 17:42:23 smithi178 systemd[1]: Starting Ceph mds.f for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:42:23.990 INFO:journalctl@ceph.mds.f.smithi178.stdout:Jul 14 17:42:23 smithi178 podman[154875]: 2023-07-14T17:42:24.252 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:42:24.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.403+0000 7f7c7dbc3700 1 -- 172.21.15.88:0/1455927371 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c780ffd10 msgr2=0x7f7c78100160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:24.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.403+0000 7f7c7dbc3700 1 --2- 172.21.15.88:0/1455927371 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c780ffd10 0x7f7c78100160 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c009990 tx=0x7f7c6c009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:24.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.403+0000 7f7c7dbc3700 1 -- 172.21.15.88:0/1455927371 shutdown_connections 2023-07-14T17:42:24.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.403+0000 7f7c7dbc3700 1 --2- 172.21.15.88:0/1455927371 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c781006a0 0x7f7c78106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:24.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.403+0000 7f7c7dbc3700 1 --2- 172.21.15.88:0/1455927371 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c780feb20 0x7f7c780fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:24.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.403+0000 7f7c7dbc3700 1 --2- 172.21.15.88:0/1455927371 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c780ffd10 0x7f7c78100160 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:24.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.403+0000 7f7c7dbc3700 1 -- 172.21.15.88:0/1455927371 >> 172.21.15.88:0/1455927371 conn(0x7f7c780fa100 msgr2=0x7f7c780fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:24.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.403+0000 7f7c7dbc3700 1 -- 172.21.15.88:0/1455927371 shutdown_connections 2023-07-14T17:42:24.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.403+0000 7f7c7dbc3700 1 -- 172.21.15.88:0/1455927371 wait complete. 2023-07-14T17:42:24.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.404+0000 7f7c7dbc3700 1 Processor -- start 2023-07-14T17:42:24.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.404+0000 7f7c7dbc3700 1 -- start start 2023-07-14T17:42:24.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.404+0000 7f7c7dbc3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c780feb20 0x7f7c78069360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:24.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.404+0000 7f7c7dbc3700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c780ffd10 0x7f7c7819a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:24.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.404+0000 7f7c7dbc3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c781006a0 0x7f7c7819cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:24.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.404+0000 7f7c7dbc3700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7819d3f0 con 0x7f7c781006a0 2023-07-14T17:42:24.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.404+0000 7f7c7dbc3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7819d560 con 0x7f7c780feb20 2023-07-14T17:42:24.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.404+0000 7f7c7dbc3700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7819d720 con 0x7f7c780ffd10 2023-07-14T17:42:24.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.404+0000 7f7c77fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c781006a0 0x7f7c7819cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:24.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.404+0000 7f7c77fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c781006a0 0x7f7c7819cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41228/0 (socket says 172.21.15.88:41228) 2023-07-14T17:42:24.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.404+0000 7f7c777fe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c780feb20 0x7f7c78069360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:24.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.404+0000 7f7c77fff700 1 -- 172.21.15.88:0/3700953448 learned_addr learned my addr 172.21.15.88:0/3700953448 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:24.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.404+0000 7f7c76ffd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c780ffd10 0x7f7c7819a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:24.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.405+0000 7f7c77fff700 1 -- 172.21.15.88:0/3700953448 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c780ffd10 msgr2=0x7f7c7819a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:24.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.405+0000 7f7c77fff700 1 --2- 172.21.15.88:0/3700953448 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c780ffd10 0x7f7c7819a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:24.391 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.405+0000 7f7c77fff700 1 -- 172.21.15.88:0/3700953448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c780feb20 msgr2=0x7f7c78069360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:24.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.405+0000 7f7c77fff700 1 --2- 172.21.15.88:0/3700953448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c780feb20 0x7f7c78069360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:24.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.405+0000 7f7c77fff700 1 -- 172.21.15.88:0/3700953448 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c6c0095f0 con 0x7f7c781006a0 2023-07-14T17:42:24.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.405+0000 7f7c77fff700 1 --2- 172.21.15.88:0/3700953448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c781006a0 0x7f7c7819cc20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000b970 tx=0x7f7c7000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:24.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.405+0000 7f7c74ff9700 1 -- 172.21.15.88:0/3700953448 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7000d860 con 0x7f7c781006a0 2023-07-14T17:42:24.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.405+0000 7f7c74ff9700 1 -- 172.21.15.88:0/3700953448 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7c7000de50 con 0x7f7c781006a0 2023-07-14T17:42:24.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.405+0000 7f7c7dbc3700 1 -- 172.21.15.88:0/3700953448 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7819da00 con 0x7f7c781006a0 2023-07-14T17:42:24.392 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.405+0000 7f7c74ff9700 1 -- 172.21.15.88:0/3700953448 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c70017400 con 0x7f7c781006a0 2023-07-14T17:42:24.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.405+0000 7f7c7dbc3700 1 -- 172.21.15.88:0/3700953448 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7819dfe0 con 0x7f7c781006a0 2023-07-14T17:42:24.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.406+0000 7f7c7dbc3700 1 -- 172.21.15.88:0/3700953448 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7810a270 con 0x7f7c781006a0 2023-07-14T17:42:24.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.407+0000 7f7c74ff9700 1 -- 172.21.15.88:0/3700953448 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7c7000d9c0 con 0x7f7c781006a0 2023-07-14T17:42:24.393 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.407+0000 7f7c74ff9700 1 --2- 172.21.15.88:0/3700953448 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7c600a1220 0x7f7c600a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:24.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.408+0000 7f7c777fe700 1 --2- 172.21.15.88:0/3700953448 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7c600a1220 0x7f7c600a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:24.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.408+0000 7f7c74ff9700 1 -- 172.21.15.88:0/3700953448 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7c70006e40 con 0x7f7c781006a0 2023-07-14T17:42:24.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.408+0000 7f7c777fe700 1 --2- 172.21.15.88:0/3700953448 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7c600a1220 0x7f7c600a36d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7c780ffb70 tx=0x7f7c6800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:24.394 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.412+0000 7f7c74ff9700 1 -- 172.21.15.88:0/3700953448 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7c70095b60 con 0x7f7c781006a0 2023-07-14T17:42:24.414 INFO:journalctl@ceph.mds.f.smithi178.stdout:Jul 14 17:42:24 smithi178 ceph-mds[154915]: -- 172.21.15.178:0/1133708098 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55acd86151e0 con 0x55acd85eac00 2023-07-14T17:42:24.414 INFO:journalctl@ceph.mds.f.smithi178.stdout:Jul 14 17:42:24 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-f[154891]: starting mds.f at 2023-07-14T17:42:24.415 INFO:journalctl@ceph.mds.f.smithi178.stdout:Jul 14 17:42:24 smithi178 bash[154875]: 1ca0ea4f0d31f4808c552074ba7ea1b58fcff5e0edd68acbd1dae17805cee570 2023-07-14T17:42:24.415 INFO:journalctl@ceph.mds.f.smithi178.stdout:Jul 14 17:42:24 smithi178 systemd[1]: Started Ceph mds.f for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:42:24.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.641+0000 7f7c7dbc3700 1 -- 172.21.15.88:0/3700953448 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c7810a460 con 0x7f7c781006a0 2023-07-14T17:42:24.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.642+0000 7f7c74ff9700 1 -- 172.21.15.88:0/3700953448 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4291 (secure 0 0 0) 0x7f7c70099180 con 0x7f7c781006a0 2023-07-14T17:42:24.625 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:24.625 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3062637672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3062637672},{"type":"v1","addr":"172.21.15.107:6835","nonce":3062637672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-07-14T17:42:24.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.644+0000 7f7c7dbc3700 1 -- 172.21.15.88:0/3700953448 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7c600a1220 msgr2=0x7f7c600a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:24.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.644+0000 7f7c7dbc3700 1 --2- 172.21.15.88:0/3700953448 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7c600a1220 0x7f7c600a36d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7c780ffb70 tx=0x7f7c6800b410 comp rx=0 tx=0).stop 2023-07-14T17:42:24.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.644+0000 7f7c7dbc3700 1 -- 172.21.15.88:0/3700953448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c781006a0 msgr2=0x7f7c7819cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:24.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.644+0000 7f7c7dbc3700 1 --2- 172.21.15.88:0/3700953448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c781006a0 0x7f7c7819cc20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7c7000b970 tx=0x7f7c7000bc80 comp rx=0 tx=0).stop 2023-07-14T17:42:24.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.644+0000 7f7c7dbc3700 1 -- 172.21.15.88:0/3700953448 shutdown_connections 2023-07-14T17:42:24.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.644+0000 7f7c7dbc3700 1 --2- 172.21.15.88:0/3700953448 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7c600a1220 0x7f7c600a36d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:24.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.644+0000 7f7c7dbc3700 1 --2- 172.21.15.88:0/3700953448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c781006a0 0x7f7c7819cc20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:24.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.644+0000 7f7c7dbc3700 1 --2- 172.21.15.88:0/3700953448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7c780feb20 0x7f7c78069360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:24.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.644+0000 7f7c7dbc3700 1 --2- 172.21.15.88:0/3700953448 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7c780ffd10 0x7f7c7819a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:24.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.644+0000 7f7c7dbc3700 1 -- 172.21.15.88:0/3700953448 >> 172.21.15.88:0/3700953448 conn(0x7f7c780fa100 msgr2=0x7f7c780fc370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:24.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.644+0000 7f7c7dbc3700 1 -- 172.21.15.88:0/3700953448 shutdown_connections 2023-07-14T17:42:24.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:24.644+0000 7f7c7dbc3700 1 -- 172.21.15.88:0/3700953448 wait complete. 2023-07-14T17:42:24.628 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 6 2023-07-14T17:42:25.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:25 smithi178 ceph-mon[129368]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:25.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:25.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:25.356 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:25.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:25.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:25.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:25.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:25 smithi178 ceph-mon[129368]: Deploying daemon mds.c on smithi178 2023-07-14T17:42:25.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:25 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3700953448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:25.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:25 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.178:6834/2332025068,v1:172.21.15.178:6835/2332025068] up:boot 2023-07-14T17:42:25.357 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:25 smithi178 ceph-mon[129368]: fsmap 6 up:standby 2023-07-14T17:42:25.358 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:25 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T17:42:25.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:25 smithi088 ceph-mon[127465]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:25.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:25.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:25.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:25.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:25.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:25.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:25.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:25 smithi088 ceph-mon[127465]: Deploying daemon mds.c on smithi178 2023-07-14T17:42:25.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:25 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3700953448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:25.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:25 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.178:6834/2332025068,v1:172.21.15.178:6835/2332025068] up:boot 2023-07-14T17:42:25.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:25 smithi088 ceph-mon[127465]: fsmap 6 up:standby 2023-07-14T17:42:25.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:25 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T17:42:25.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:25 smithi107 ceph-mon[128999]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:25.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:25 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:25.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:25 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:25.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:25 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:25.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:25 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:25.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:25 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:25.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:25 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:25.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:25 smithi107 ceph-mon[128999]: Deploying daemon mds.c on smithi178 2023-07-14T17:42:25.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:25 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3700953448' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:25.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:25 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.178:6834/2332025068,v1:172.21.15.178:6835/2332025068] up:boot 2023-07-14T17:42:25.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:25 smithi107 ceph-mon[128999]: fsmap 6 up:standby 2023-07-14T17:42:25.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:25 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T17:42:25.915 INFO:journalctl@ceph.mds.c.smithi178.stdout:Jul 14 17:42:25 smithi178 systemd[1]: Starting Ceph mds.c for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:42:26.350 INFO:journalctl@ceph.mds.c.smithi178.stdout:Jul 14 17:42:26 smithi178 podman[155227]: 2023-07-14T17:42:26.639 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:42:26.664 INFO:journalctl@ceph.mds.c.smithi178.stdout:Jul 14 17:42:26 smithi178 ceph-mds[155268]: -- 172.21.15.178:0/1129749572 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560092b671e0 con 0x560092b3cc00 2023-07-14T17:42:26.664 INFO:journalctl@ceph.mds.c.smithi178.stdout:Jul 14 17:42:26 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-c[155243]: starting mds.c at 2023-07-14T17:42:26.665 INFO:journalctl@ceph.mds.c.smithi178.stdout:Jul 14 17:42:26 smithi178 bash[155227]: 215025bb1f805f2221ad913b6d666f3ed9b32b2f99f64d6d1315836ce2c8c0f7 2023-07-14T17:42:26.665 INFO:journalctl@ceph.mds.c.smithi178.stdout:Jul 14 17:42:26 smithi178 systemd[1]: Started Ceph mds.c for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:42:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.790+0000 7fa1cfafe700 1 -- 172.21.15.88:0/1542792784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1c8102fc0 msgr2=0x7fa1c81053a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.790+0000 7fa1cfafe700 1 --2- 172.21.15.88:0/1542792784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1c8102fc0 0x7fa1c81053a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4009990 tx=0x7fa1c4009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:26.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.790+0000 7fa1cfafe700 1 -- 172.21.15.88:0/1542792784 shutdown_connections 2023-07-14T17:42:26.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.790+0000 7fa1cfafe700 1 --2- 172.21.15.88:0/1542792784 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1c81058e0 0x7fa1c8107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:26.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.790+0000 7fa1cfafe700 1 --2- 172.21.15.88:0/1542792784 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1c81006a0 0x7fa1c8102a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:26.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.790+0000 7fa1cfafe700 1 --2- 172.21.15.88:0/1542792784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1c8102fc0 0x7fa1c81053a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:26.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.790+0000 7fa1cfafe700 1 -- 172.21.15.88:0/1542792784 >> 172.21.15.88:0/1542792784 conn(0x7fa1c80fa100 msgr2=0x7fa1c80fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:26.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.790+0000 7fa1cfafe700 1 -- 172.21.15.88:0/1542792784 shutdown_connections 2023-07-14T17:42:26.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.790+0000 7fa1cfafe700 1 -- 172.21.15.88:0/1542792784 wait complete. 2023-07-14T17:42:26.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1cfafe700 1 Processor -- start 2023-07-14T17:42:26.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1cfafe700 1 -- start start 2023-07-14T17:42:26.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1cfafe700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1c81006a0 0x7fa1c8100010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:26.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1cfafe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1c8102fc0 0x7fa1c80fe650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:26.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1cfafe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1c81058e0 0x7fa1c80feb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:26.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1cfafe700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c80ff330 con 0x7fa1c8102fc0 2023-07-14T17:42:26.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1cfafe700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c80ff4a0 con 0x7fa1c81058e0 2023-07-14T17:42:26.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1cfafe700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c819f8f0 con 0x7fa1c81006a0 2023-07-14T17:42:26.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1cd099700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1c8102fc0 0x7fa1c80fe650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:26.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1cd89a700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1c81006a0 0x7fa1c8100010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:26.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1cd099700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1c8102fc0 0x7fa1c80fe650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41240/0 (socket says 172.21.15.88:41240) 2023-07-14T17:42:26.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1cd099700 1 -- 172.21.15.88:0/264745353 learned_addr learned my addr 172.21.15.88:0/264745353 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:26.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1ce09b700 1 --2- 172.21.15.88:0/264745353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1c81058e0 0x7fa1c80feb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:26.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1cd099700 1 -- 172.21.15.88:0/264745353 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1c81006a0 msgr2=0x7fa1c8100010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:26.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1cd099700 1 --2- 172.21.15.88:0/264745353 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1c81006a0 0x7fa1c8100010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:26.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1cd099700 1 -- 172.21.15.88:0/264745353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1c81058e0 msgr2=0x7fa1c80feb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:26.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1cd099700 1 --2- 172.21.15.88:0/264745353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1c81058e0 0x7fa1c80feb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:26.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1cd099700 1 -- 172.21.15.88:0/264745353 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c40095f0 con 0x7fa1c8102fc0 2023-07-14T17:42:26.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1cd099700 1 --2- 172.21.15.88:0/264745353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1c8102fc0 0x7fa1c80fe650 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4005910 tx=0x7fa1c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:26.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.791+0000 7fa1beffd700 1 -- 172.21.15.88:0/264745353 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c401d070 con 0x7fa1c8102fc0 2023-07-14T17:42:26.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.792+0000 7fa1beffd700 1 -- 172.21.15.88:0/264745353 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa1c4004c70 con 0x7fa1c8102fc0 2023-07-14T17:42:26.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.792+0000 7fa1cfafe700 1 -- 172.21.15.88:0/264745353 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c819fb70 con 0x7fa1c8102fc0 2023-07-14T17:42:26.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.792+0000 7fa1beffd700 1 -- 172.21.15.88:0/264745353 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4003a30 con 0x7fa1c8102fc0 2023-07-14T17:42:26.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.792+0000 7fa1cfafe700 1 -- 172.21.15.88:0/264745353 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c819ff90 con 0x7fa1c8102fc0 2023-07-14T17:42:26.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.792+0000 7fa1cfafe700 1 -- 172.21.15.88:0/264745353 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1c810b350 con 0x7fa1c8102fc0 2023-07-14T17:42:26.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.794+0000 7fa1beffd700 1 -- 172.21.15.88:0/264745353 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa1c4005000 con 0x7fa1c8102fc0 2023-07-14T17:42:26.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.795+0000 7fa1beffd700 1 --2- 172.21.15.88:0/264745353 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa1b40a11a0 0x7fa1b40a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:26.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.795+0000 7fa1cd89a700 1 --2- 172.21.15.88:0/264745353 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa1b40a11a0 0x7fa1b40a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:26.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.795+0000 7fa1beffd700 1 -- 172.21.15.88:0/264745353 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa1c40cd750 con 0x7fa1c8102fc0 2023-07-14T17:42:26.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.795+0000 7fa1cd89a700 1 --2- 172.21.15.88:0/264745353 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa1b40a11a0 0x7fa1b40a3650 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8009a10 tx=0x7fa1b8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:26.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:26.798+0000 7fa1beffd700 1 -- 172.21.15.88:0/264745353 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa1c409ae60 con 0x7fa1c8102fc0 2023-07-14T17:42:26.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:26 smithi088 ceph-mon[127465]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:26.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:26 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:26.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:26 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:26.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:26 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:26.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:26 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:26.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:26 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:26.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:26 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:26.974 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:26 smithi107 ceph-mon[128999]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:26.974 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:26 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:26.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:26 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:26.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:26 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:26.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:26 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:26.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:26 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:26.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:26 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:27.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:27.031+0000 7fa1cfafe700 1 -- 172.21.15.88:0/264745353 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1c81a0660 con 0x7fa1c8102fc0 2023-07-14T17:42:27.014 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:27.032+0000 7fa1beffd700 1 -- 172.21.15.88:0/264745353 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5811 (secure 0 0 0) 0x7fa1c409a9f0 con 0x7fa1c8102fc0 2023-07-14T17:42:27.015 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:27.015 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/2332025068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":2332025068},{"type":"v1","addr":"172.21.15.178:6835","nonce":2332025068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3062637672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3062637672},{"type":"v1","addr":"172.21.15.107:6835","nonce":3062637672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/3950404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3950404084},{"type":"v1","addr":"172.21.15.178:6837","nonce":3950404084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-14T17:42:27.016 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:27.034+0000 7fa1cfafe700 1 -- 172.21.15.88:0/264745353 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa1b40a11a0 msgr2=0x7fa1b40a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:27.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:27.034+0000 7fa1cfafe700 1 --2- 172.21.15.88:0/264745353 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa1b40a11a0 0x7fa1b40a3650 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8009a10 tx=0x7fa1b8009190 comp rx=0 tx=0).stop 2023-07-14T17:42:27.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:27.034+0000 7fa1cfafe700 1 -- 172.21.15.88:0/264745353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1c8102fc0 msgr2=0x7fa1c80fe650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:27.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:27.034+0000 7fa1cfafe700 1 --2- 172.21.15.88:0/264745353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1c8102fc0 0x7fa1c80fe650 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4005910 tx=0x7fa1c4004730 comp rx=0 tx=0).stop 2023-07-14T17:42:27.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:27.034+0000 7fa1cfafe700 1 -- 172.21.15.88:0/264745353 shutdown_connections 2023-07-14T17:42:27.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:27.034+0000 7fa1cfafe700 1 --2- 172.21.15.88:0/264745353 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa1c81058e0 0x7fa1c80feb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:27.017 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:27.034+0000 7fa1cfafe700 1 --2- 172.21.15.88:0/264745353 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa1c81006a0 0x7fa1c8100010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:27.034+0000 7fa1cfafe700 1 --2- 172.21.15.88:0/264745353 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa1b40a11a0 0x7fa1b40a3650 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:27.034+0000 7fa1cfafe700 1 --2- 172.21.15.88:0/264745353 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa1c8102fc0 0x7fa1c80fe650 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:27.034+0000 7fa1cfafe700 1 -- 172.21.15.88:0/264745353 >> 172.21.15.88:0/264745353 conn(0x7fa1c80fa100 msgr2=0x7fa1c8101b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:27.034+0000 7fa1cfafe700 1 -- 172.21.15.88:0/264745353 shutdown_connections 2023-07-14T17:42:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:27.034+0000 7fa1cfafe700 1 -- 172.21.15.88:0/264745353 wait complete. 2023-07-14T17:42:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 8 2023-07-14T17:42:27.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:26 smithi178 ceph-mon[129368]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:27.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:26 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:27.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:26 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:27.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:26 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:27.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:26 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:27.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:26 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:27.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:26 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:27.771 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:27 smithi107 ceph-mon[128999]: Deploying daemon mds.h on smithi107 2023-07-14T17:42:27.772 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:27 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] up:boot 2023-07-14T17:42:27.772 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:27 smithi107 ceph-mon[128999]: fsmap 7 up:standby 2023-07-14T17:42:27.772 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:27 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T17:42:27.772 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:27 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/264745353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:27.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:27 smithi088 ceph-mon[127465]: Deploying daemon mds.h on smithi107 2023-07-14T17:42:27.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:27 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] up:boot 2023-07-14T17:42:27.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:27 smithi088 ceph-mon[127465]: fsmap 7 up:standby 2023-07-14T17:42:27.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:27 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T17:42:27.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:27 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/264745353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:28.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:27 smithi178 ceph-mon[129368]: Deploying daemon mds.h on smithi107 2023-07-14T17:42:28.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:27 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] up:boot 2023-07-14T17:42:28.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:27 smithi178 ceph-mon[129368]: fsmap 7 up:standby 2023-07-14T17:42:28.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:27 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T17:42:28.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:27 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/264745353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:28.440 INFO:journalctl@ceph.mds.h.smithi107.stdout:Jul 14 17:42:28 smithi107 systemd[1]: Starting Ceph mds.h for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:42:28.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:28 smithi107 ceph-mon[128999]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:28.715 INFO:journalctl@ceph.mds.h.smithi107.stdout:Jul 14 17:42:28 smithi107 podman[153724]: 2023-07-14T17:42:28.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:28 smithi088 ceph-mon[127465]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:28.974 INFO:journalctl@ceph.mds.h.smithi107.stdout:Jul 14 17:42:28 smithi107 ceph-mds[153764]: -- 172.21.15.107:0/3238453736 <== mon.2 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ee07a571e0 con 0x55ee07a2cc00 2023-07-14T17:42:28.975 INFO:journalctl@ceph.mds.h.smithi107.stdout:Jul 14 17:42:28 smithi107 bash[153724]: b81cdf5c404c88340313b7ceeefc03dc54055b101bf1ebd6fbbca7e09dbba35c 2023-07-14T17:42:28.975 INFO:journalctl@ceph.mds.h.smithi107.stdout:Jul 14 17:42:28 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-h[153740]: starting mds.h at 2023-07-14T17:42:28.975 INFO:journalctl@ceph.mds.h.smithi107.stdout:Jul 14 17:42:28 smithi107 systemd[1]: Started Ceph mds.h for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:42:29.029 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:42:29.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.180+0000 7fce21dfa700 1 -- 172.21.15.88:0/891595127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce1c100680 msgr2=0x7fce1c106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:29.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.180+0000 7fce21dfa700 1 --2- 172.21.15.88:0/891595127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce1c100680 0x7fce1c106f40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fce1400b0a0 tx=0x7fce1400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:29.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.180+0000 7fce21dfa700 1 -- 172.21.15.88:0/891595127 shutdown_connections 2023-07-14T17:42:29.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.180+0000 7fce21dfa700 1 --2- 172.21.15.88:0/891595127 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce1c100680 0x7fce1c106f40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:29.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.180+0000 7fce21dfa700 1 --2- 172.21.15.88:0/891595127 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce1c0feb00 0x7fce1c0fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:29.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.180+0000 7fce21dfa700 1 --2- 172.21.15.88:0/891595127 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1c0ffcf0 0x7fce1c100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:29.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.180+0000 7fce21dfa700 1 -- 172.21.15.88:0/891595127 >> 172.21.15.88:0/891595127 conn(0x7fce1c0fa100 msgr2=0x7fce1c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:29.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.180+0000 7fce21dfa700 1 -- 172.21.15.88:0/891595127 shutdown_connections 2023-07-14T17:42:29.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.181+0000 7fce21dfa700 1 -- 172.21.15.88:0/891595127 wait complete. 2023-07-14T17:42:29.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.181+0000 7fce21dfa700 1 Processor -- start 2023-07-14T17:42:29.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:28 smithi178 ceph-mon[129368]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:29.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.181+0000 7fce21dfa700 1 -- start start 2023-07-14T17:42:29.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.181+0000 7fce21dfa700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce1c0feb00 0x7fce1c195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:29.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.181+0000 7fce21dfa700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce1c0ffcf0 0x7fce1c195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:29.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.181+0000 7fce21dfa700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1c100680 0x7fce1c19cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:29.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.181+0000 7fce21dfa700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c19d510 con 0x7fce1c0ffcf0 2023-07-14T17:42:29.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.181+0000 7fce21dfa700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c19d650 con 0x7fce1c100680 2023-07-14T17:42:29.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.181+0000 7fce21dfa700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1c19d7e0 con 0x7fce1c0feb00 2023-07-14T17:42:29.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.182+0000 7fce1affd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce1c0ffcf0 0x7fce1c195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:29.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.182+0000 7fce1affd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce1c0ffcf0 0x7fce1c195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41252/0 (socket says 172.21.15.88:41252) 2023-07-14T17:42:29.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.182+0000 7fce1affd700 1 -- 172.21.15.88:0/963685975 learned_addr learned my addr 172.21.15.88:0/963685975 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:29.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.182+0000 7fce1bfff700 1 --2- 172.21.15.88:0/963685975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1c100680 0x7fce1c19cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:29.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.182+0000 7fce1b7fe700 1 --2- 172.21.15.88:0/963685975 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce1c0feb00 0x7fce1c195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:29.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.182+0000 7fce1affd700 1 -- 172.21.15.88:0/963685975 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce1c0feb00 msgr2=0x7fce1c195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:29.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.182+0000 7fce1affd700 1 --2- 172.21.15.88:0/963685975 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce1c0feb00 0x7fce1c195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:29.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.182+0000 7fce1affd700 1 -- 172.21.15.88:0/963685975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1c100680 msgr2=0x7fce1c19cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:29.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.182+0000 7fce1affd700 1 --2- 172.21.15.88:0/963685975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1c100680 0x7fce1c19cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:29.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.182+0000 7fce1affd700 1 -- 172.21.15.88:0/963685975 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce14009d00 con 0x7fce1c0ffcf0 2023-07-14T17:42:29.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.182+0000 7fce1b7fe700 1 --2- 172.21.15.88:0/963685975 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce1c0feb00 0x7fce1c195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:42:29.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.182+0000 7fce1affd700 1 --2- 172.21.15.88:0/963685975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce1c0ffcf0 0x7fce1c195ed0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fce1000d950 tx=0x7fce1000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:29.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.182+0000 7fce18ff9700 1 -- 172.21.15.88:0/963685975 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce10009940 con 0x7fce1c0ffcf0 2023-07-14T17:42:29.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.182+0000 7fce18ff9700 1 -- 172.21.15.88:0/963685975 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fce10010460 con 0x7fce1c0ffcf0 2023-07-14T17:42:29.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.182+0000 7fce21dfa700 1 -- 172.21.15.88:0/963685975 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce1c19dac0 con 0x7fce1c0ffcf0 2023-07-14T17:42:29.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.182+0000 7fce18ff9700 1 -- 172.21.15.88:0/963685975 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1000f620 con 0x7fce1c0ffcf0 2023-07-14T17:42:29.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.182+0000 7fce21dfa700 1 -- 172.21.15.88:0/963685975 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce1c19e020 con 0x7fce1c0ffcf0 2023-07-14T17:42:29.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.183+0000 7fce21dfa700 1 -- 172.21.15.88:0/963685975 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1c106be0 con 0x7fce1c0ffcf0 2023-07-14T17:42:29.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.184+0000 7fce18ff9700 1 -- 172.21.15.88:0/963685975 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fce100105d0 con 0x7fce1c0ffcf0 2023-07-14T17:42:29.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.185+0000 7fce18ff9700 1 --2- 172.21.15.88:0/963685975 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fce040a1220 0x7fce040a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:29.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.185+0000 7fce1b7fe700 1 --2- 172.21.15.88:0/963685975 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fce040a1220 0x7fce040a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:29.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.185+0000 7fce18ff9700 1 -- 172.21.15.88:0/963685975 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fce100cc220 con 0x7fce1c0ffcf0 2023-07-14T17:42:29.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.185+0000 7fce1b7fe700 1 --2- 172.21.15.88:0/963685975 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fce040a1220 0x7fce040a36d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fce1c0ffb50 tx=0x7fce0c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:29.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.189+0000 7fce18ff9700 1 -- 172.21.15.88:0/963685975 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fce10095e10 con 0x7fce1c0ffcf0 2023-07-14T17:42:29.406 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.424+0000 7fce21dfa700 1 -- 172.21.15.88:0/963685975 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce1c02c8e0 con 0x7fce1c0ffcf0 2023-07-14T17:42:29.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.424+0000 7fce18ff9700 1 -- 172.21.15.88:0/963685975 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5811 (secure 0 0 0) 0x7fce10099430 con 0x7fce1c0ffcf0 2023-07-14T17:42:29.407 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:29.407 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/2332025068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":2332025068},{"type":"v1","addr":"172.21.15.178:6835","nonce":2332025068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3062637672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3062637672},{"type":"v1","addr":"172.21.15.107:6835","nonce":3062637672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/3950404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3950404084},{"type":"v1","addr":"172.21.15.178:6837","nonce":3950404084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-14T17:42:29.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.426+0000 7fce21dfa700 1 -- 172.21.15.88:0/963685975 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fce040a1220 msgr2=0x7fce040a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:29.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.426+0000 7fce21dfa700 1 --2- 172.21.15.88:0/963685975 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fce040a1220 0x7fce040a36d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fce1c0ffb50 tx=0x7fce0c009260 comp rx=0 tx=0).stop 2023-07-14T17:42:29.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.426+0000 7fce21dfa700 1 -- 172.21.15.88:0/963685975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce1c0ffcf0 msgr2=0x7fce1c195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:29.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.426+0000 7fce21dfa700 1 --2- 172.21.15.88:0/963685975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce1c0ffcf0 0x7fce1c195ed0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fce1000d950 tx=0x7fce1000dc60 comp rx=0 tx=0).stop 2023-07-14T17:42:29.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.426+0000 7fce21dfa700 1 -- 172.21.15.88:0/963685975 shutdown_connections 2023-07-14T17:42:29.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.426+0000 7fce21dfa700 1 --2- 172.21.15.88:0/963685975 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fce1c100680 0x7fce1c19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:29.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.426+0000 7fce21dfa700 1 --2- 172.21.15.88:0/963685975 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fce1c0feb00 0x7fce1c195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:29.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.426+0000 7fce21dfa700 1 --2- 172.21.15.88:0/963685975 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fce040a1220 0x7fce040a36d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:29.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.426+0000 7fce21dfa700 1 --2- 172.21.15.88:0/963685975 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce1c0ffcf0 0x7fce1c195ed0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:29.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.427+0000 7fce21dfa700 1 -- 172.21.15.88:0/963685975 >> 172.21.15.88:0/963685975 conn(0x7fce1c0fa100 msgr2=0x7fce1c0fc310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:29.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.427+0000 7fce21dfa700 1 -- 172.21.15.88:0/963685975 shutdown_connections 2023-07-14T17:42:29.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:29.427+0000 7fce21dfa700 1 -- 172.21.15.88:0/963685975 wait complete. 2023-07-14T17:42:29.411 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 8 2023-07-14T17:42:30.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:29 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:30.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:29 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:30.121 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:29 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:30.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:29 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:30.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:29 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:30.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:29 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:30.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:29 smithi178 ceph-mon[129368]: Deploying daemon mds.i on smithi178 2023-07-14T17:42:30.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:29 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/963685975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:30.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:29 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] up:boot 2023-07-14T17:42:30.122 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:29 smithi178 ceph-mon[129368]: fsmap 8 up:standby 2023-07-14T17:42:30.123 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:29 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T17:42:30.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:29 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:30.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:29 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:30.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:29 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:30.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:29 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:30.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:29 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:30.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:29 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:30.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:29 smithi088 ceph-mon[127465]: Deploying daemon mds.i on smithi178 2023-07-14T17:42:30.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:29 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/963685975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:30.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:29 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] up:boot 2023-07-14T17:42:30.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:29 smithi088 ceph-mon[127465]: fsmap 8 up:standby 2023-07-14T17:42:30.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:29 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T17:42:30.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:29 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:30.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:29 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:30.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:29 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:30.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:29 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:30.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:29 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:30.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:29 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:30.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:29 smithi107 ceph-mon[128999]: Deploying daemon mds.i on smithi178 2023-07-14T17:42:30.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:29 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/963685975' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:30.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:29 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] up:boot 2023-07-14T17:42:30.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:29 smithi107 ceph-mon[128999]: fsmap 8 up:standby 2023-07-14T17:42:30.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:29 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T17:42:30.759 INFO:journalctl@ceph.mds.i.smithi178.stdout:Jul 14 17:42:30 smithi178 systemd[1]: Starting Ceph mds.i for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:42:31.011 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:30 smithi178 ceph-mon[129368]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:31.012 INFO:journalctl@ceph.mds.i.smithi178.stdout:Jul 14 17:42:30 smithi178 podman[155567]: 2023-07-14T17:42:31.012 INFO:journalctl@ceph.mds.i.smithi178.stdout:Jul 14 17:42:31 smithi178 bash[155567]: 57ec7a569420172d4dbd40e464f5bbca524a85cae2ab0cf188e8a2e5ef962b28 2023-07-14T17:42:31.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:30 smithi088 ceph-mon[127465]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:31.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:30 smithi107 ceph-mon[128999]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:31.414 INFO:journalctl@ceph.mds.i.smithi178.stdout:Jul 14 17:42:31 smithi178 ceph-mds[155607]: -- 172.21.15.178:0/2325675259 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5613d6fb71e0 con 0x5613d6f8c800 2023-07-14T17:42:31.414 INFO:journalctl@ceph.mds.i.smithi178.stdout:Jul 14 17:42:31 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-i[155583]: starting mds.i at 2023-07-14T17:42:31.415 INFO:journalctl@ceph.mds.i.smithi178.stdout:Jul 14 17:42:31 smithi178 systemd[1]: Started Ceph mds.i for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:42:31.422 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:42:31.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.573+0000 7f56cb8f5700 1 -- 172.21.15.88:0/96971456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56c40ffcf0 msgr2=0x7f56c4100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:31.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.573+0000 7f56cb8f5700 1 --2- 172.21.15.88:0/96971456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56c40ffcf0 0x7f56c4100140 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f56b4009990 tx=0x7f56b4009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:31.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.573+0000 7f56cb8f5700 1 -- 172.21.15.88:0/96971456 shutdown_connections 2023-07-14T17:42:31.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.573+0000 7f56cb8f5700 1 --2- 172.21.15.88:0/96971456 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56c4100680 0x7f56c4106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:31.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.573+0000 7f56cb8f5700 1 --2- 172.21.15.88:0/96971456 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56c40feb00 0x7f56c40fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:31.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.573+0000 7f56cb8f5700 1 --2- 172.21.15.88:0/96971456 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56c40ffcf0 0x7f56c4100140 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:31.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.573+0000 7f56cb8f5700 1 -- 172.21.15.88:0/96971456 >> 172.21.15.88:0/96971456 conn(0x7f56c40fa100 msgr2=0x7f56c40fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:31.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.573+0000 7f56cb8f5700 1 -- 172.21.15.88:0/96971456 shutdown_connections 2023-07-14T17:42:31.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.573+0000 7f56cb8f5700 1 -- 172.21.15.88:0/96971456 wait complete. 2023-07-14T17:42:31.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.574+0000 7f56cb8f5700 1 Processor -- start 2023-07-14T17:42:31.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.574+0000 7f56cb8f5700 1 -- start start 2023-07-14T17:42:31.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.574+0000 7f56cb8f5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56c40feb00 0x7f56c4195840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:31.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.574+0000 7f56cb8f5700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56c40ffcf0 0x7f56c4195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:31.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.574+0000 7f56cb8f5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56c4100680 0x7f56c419cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:31.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.574+0000 7f56cb8f5700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c419d3c0 con 0x7f56c40feb00 2023-07-14T17:42:31.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.574+0000 7f56cb8f5700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c419d500 con 0x7f56c4100680 2023-07-14T17:42:31.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.574+0000 7f56cb8f5700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c419d690 con 0x7f56c40ffcf0 2023-07-14T17:42:31.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.574+0000 7f56c9691700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56c40feb00 0x7f56c4195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:31.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.574+0000 7f56c9691700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56c40feb00 0x7f56c4195840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41260/0 (socket says 172.21.15.88:41260) 2023-07-14T17:42:31.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.574+0000 7f56c9691700 1 -- 172.21.15.88:0/1071586119 learned_addr learned my addr 172.21.15.88:0/1071586119 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:31.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.574+0000 7f56c8e90700 1 --2- 172.21.15.88:0/1071586119 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56c40ffcf0 0x7f56c4195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:31.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.574+0000 7f56c9e92700 1 --2- 172.21.15.88:0/1071586119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56c4100680 0x7f56c419cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:31.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.575+0000 7f56c9691700 1 -- 172.21.15.88:0/1071586119 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56c40ffcf0 msgr2=0x7f56c4195d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:31.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.575+0000 7f56c9691700 1 --2- 172.21.15.88:0/1071586119 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56c40ffcf0 0x7f56c4195d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:31.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.575+0000 7f56c9691700 1 -- 172.21.15.88:0/1071586119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56c4100680 msgr2=0x7f56c419cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:31.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.575+0000 7f56c9691700 1 --2- 172.21.15.88:0/1071586119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56c4100680 0x7f56c419cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:31.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.575+0000 7f56c9691700 1 -- 172.21.15.88:0/1071586119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b40095f0 con 0x7f56c40feb00 2023-07-14T17:42:31.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.575+0000 7f56c9691700 1 --2- 172.21.15.88:0/1071586119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56c40feb00 0x7f56c4195840 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f56c000d980 tx=0x7f56c000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:31.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.575+0000 7f56ba7fc700 1 -- 172.21.15.88:0/1071586119 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c000f890 con 0x7f56c40feb00 2023-07-14T17:42:31.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.575+0000 7f56ba7fc700 1 -- 172.21.15.88:0/1071586119 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f56c000fe80 con 0x7f56c40feb00 2023-07-14T17:42:31.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.575+0000 7f56cb8f5700 1 -- 172.21.15.88:0/1071586119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c419d970 con 0x7f56c40feb00 2023-07-14T17:42:31.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.575+0000 7f56ba7fc700 1 -- 172.21.15.88:0/1071586119 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c0019360 con 0x7f56c40feb00 2023-07-14T17:42:31.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.575+0000 7f56cb8f5700 1 -- 172.21.15.88:0/1071586119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c419de80 con 0x7f56c40feb00 2023-07-14T17:42:31.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.576+0000 7f56cb8f5700 1 -- 172.21.15.88:0/1071586119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c410b350 con 0x7f56c40feb00 2023-07-14T17:42:31.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.577+0000 7f56ba7fc700 1 -- 172.21.15.88:0/1071586119 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f56c000f9f0 con 0x7f56c40feb00 2023-07-14T17:42:31.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.578+0000 7f56ba7fc700 1 --2- 172.21.15.88:0/1071586119 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f56b00a11a0 0x7f56b00a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:31.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.578+0000 7f56c8e90700 1 --2- 172.21.15.88:0/1071586119 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f56b00a11a0 0x7f56b00a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:31.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.578+0000 7f56ba7fc700 1 -- 172.21.15.88:0/1071586119 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f56c00cc700 con 0x7f56c40feb00 2023-07-14T17:42:31.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.578+0000 7f56c8e90700 1 --2- 172.21.15.88:0/1071586119 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f56b00a11a0 0x7f56b00a3650 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f56b400c010 tx=0x7f56b4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:31.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.584+0000 7f56ba7fc700 1 -- 172.21.15.88:0/1071586119 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f56c00962f0 con 0x7f56c40feb00 2023-07-14T17:42:31.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.833+0000 7f56cb8f5700 1 -- 172.21.15.88:0/1071586119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56c41059d0 con 0x7f56c40feb00 2023-07-14T17:42:31.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.834+0000 7f56ba7fc700 1 -- 172.21.15.88:0/1071586119 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6571 (secure 0 0 0) 0x7f56c0099910 con 0x7f56c40feb00 2023-07-14T17:42:31.816 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:31.816 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/2332025068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":2332025068},{"type":"v1","addr":"172.21.15.178:6835","nonce":2332025068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3062637672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3062637672},{"type":"v1","addr":"172.21.15.107:6835","nonce":3062637672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/3950404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3950404084},{"type":"v1","addr":"172.21.15.178:6837","nonce":3950404084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1152608615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1152608615},{"type":"v1","addr":"172.21.15.107:6837","nonce":1152608615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-07-14T17:42:31.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.836+0000 7f56cb8f5700 1 -- 172.21.15.88:0/1071586119 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f56b00a11a0 msgr2=0x7f56b00a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:31.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.836+0000 7f56cb8f5700 1 --2- 172.21.15.88:0/1071586119 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f56b00a11a0 0x7f56b00a3650 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f56b400c010 tx=0x7f56b4005fb0 comp rx=0 tx=0).stop 2023-07-14T17:42:31.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.836+0000 7f56cb8f5700 1 -- 172.21.15.88:0/1071586119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56c40feb00 msgr2=0x7f56c4195840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:31.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.836+0000 7f56cb8f5700 1 --2- 172.21.15.88:0/1071586119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56c40feb00 0x7f56c4195840 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f56c000d980 tx=0x7f56c000dc90 comp rx=0 tx=0).stop 2023-07-14T17:42:31.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.836+0000 7f56cb8f5700 1 -- 172.21.15.88:0/1071586119 shutdown_connections 2023-07-14T17:42:31.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.836+0000 7f56cb8f5700 1 --2- 172.21.15.88:0/1071586119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f56c4100680 0x7f56c419cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:31.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.836+0000 7f56cb8f5700 1 --2- 172.21.15.88:0/1071586119 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f56b00a11a0 0x7f56b00a3650 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:31.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.836+0000 7f56cb8f5700 1 --2- 172.21.15.88:0/1071586119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f56c40feb00 0x7f56c4195840 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:31.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.836+0000 7f56cb8f5700 1 --2- 172.21.15.88:0/1071586119 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f56c40ffcf0 0x7f56c4195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:31.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.836+0000 7f56cb8f5700 1 -- 172.21.15.88:0/1071586119 >> 172.21.15.88:0/1071586119 conn(0x7f56c40fa100 msgr2=0x7f56c40fc340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:31.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.836+0000 7f56cb8f5700 1 -- 172.21.15.88:0/1071586119 shutdown_connections 2023-07-14T17:42:31.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:31.836+0000 7f56cb8f5700 1 -- 172.21.15.88:0/1071586119 wait complete. 2023-07-14T17:42:31.820 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 9 2023-07-14T17:42:32.380 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:32.380 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:32.380 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:32.381 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:32.381 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:32.381 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:32.381 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:32 smithi088 ceph-mon[127465]: Deploying daemon mds.g on smithi088 2023-07-14T17:42:32.381 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:32 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1071586119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:32.381 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:32 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] up:boot 2023-07-14T17:42:32.381 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:32 smithi088 ceph-mon[127465]: fsmap 9 up:standby 2023-07-14T17:42:32.382 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:32 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T17:42:32.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:32.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:32.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:32.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:32.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:32.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:32.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:32 smithi178 ceph-mon[129368]: Deploying daemon mds.g on smithi088 2023-07-14T17:42:32.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:32 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1071586119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:32.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:32 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] up:boot 2023-07-14T17:42:32.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:32 smithi178 ceph-mon[129368]: fsmap 9 up:standby 2023-07-14T17:42:32.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:32 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T17:42:32.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:32.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:32.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:32.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:32.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:32.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:32.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:32 smithi107 ceph-mon[128999]: Deploying daemon mds.g on smithi088 2023-07-14T17:42:32.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:32 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1071586119' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:32.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:32 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] up:boot 2023-07-14T17:42:32.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:32 smithi107 ceph-mon[128999]: fsmap 9 up:standby 2023-07-14T17:42:32.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:32 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T17:42:32.692 INFO:journalctl@ceph.mds.g.smithi088.stdout:Jul 14 17:42:32 smithi088 systemd[1]: Starting Ceph mds.g for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:42:33.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:33 smithi088 ceph-mon[127465]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:33.192 INFO:journalctl@ceph.mds.g.smithi088.stdout:Jul 14 17:42:32 smithi088 podman[161713]: 2023-07-14T17:42:33.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:33 smithi178 ceph-mon[129368]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:33.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:33 smithi107 ceph-mon[128999]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:33.692 INFO:journalctl@ceph.mds.g.smithi088.stdout:Jul 14 17:42:33 smithi088 ceph-mds[161753]: -- 172.21.15.88:0/1097066239 <== mon.0 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bdd26791e0 con 0x55bdd264e800 2023-07-14T17:42:33.693 INFO:journalctl@ceph.mds.g.smithi088.stdout:Jul 14 17:42:33 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-g[161729]: starting mds.g at 2023-07-14T17:42:33.693 INFO:journalctl@ceph.mds.g.smithi088.stdout:Jul 14 17:42:33 smithi088 bash[161713]: 6676d3bfb7e55e20f59c396779f06446a955e1a09a01d2d860ead53afb185212 2023-07-14T17:42:33.693 INFO:journalctl@ceph.mds.g.smithi088.stdout:Jul 14 17:42:33 smithi088 systemd[1]: Started Ceph mds.g for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:42:33.831 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:42:33.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.976+0000 7fe33a328700 1 -- 172.21.15.88:0/507443791 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3340fed50 msgr2=0x7fe3340ff1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:33.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.976+0000 7fe33a328700 1 --2- 172.21.15.88:0/507443791 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3340fed50 0x7fe3340ff1c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe31c009990 tx=0x7fe31c009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:33.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.976+0000 7fe33a328700 1 -- 172.21.15.88:0/507443791 shutdown_connections 2023-07-14T17:42:33.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.976+0000 7fe33a328700 1 --2- 172.21.15.88:0/507443791 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe3340ff700 0x7fe334106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:33.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.976+0000 7fe33a328700 1 --2- 172.21.15.88:0/507443791 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe334100100 0x7fe334100510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:33.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.976+0000 7fe33a328700 1 --2- 172.21.15.88:0/507443791 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3340fed50 0x7fe3340ff1c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:33.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.976+0000 7fe33a328700 1 -- 172.21.15.88:0/507443791 >> 172.21.15.88:0/507443791 conn(0x7fe3340f98d0 msgr2=0x7fe3340fbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:33.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.976+0000 7fe33a328700 1 -- 172.21.15.88:0/507443791 shutdown_connections 2023-07-14T17:42:33.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.976+0000 7fe33a328700 1 -- 172.21.15.88:0/507443791 wait complete. 2023-07-14T17:42:33.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.976+0000 7fe33a328700 1 Processor -- start 2023-07-14T17:42:33.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe33a328700 1 -- start start 2023-07-14T17:42:33.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe33a328700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3340fed50 0x7fe3340fe7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:33.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe33a328700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3340ff700 0x7fe33419a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:33.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe33a328700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe334100100 0x7fe33419cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:33.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe33a328700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33419d300 con 0x7fe3340ff700 2023-07-14T17:42:33.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe33a328700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33419d470 con 0x7fe334100100 2023-07-14T17:42:33.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe33a328700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33419d630 con 0x7fe3340fed50 2023-07-14T17:42:33.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe3337fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3340ff700 0x7fe33419a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:33.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe3337fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3340ff700 0x7fe33419a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41284/0 (socket says 172.21.15.88:41284) 2023-07-14T17:42:33.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe3337fe700 1 -- 172.21.15.88:0/1370891029 learned_addr learned my addr 172.21.15.88:0/1370891029 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:33.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe333fff700 1 --2- 172.21.15.88:0/1370891029 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3340fed50 0x7fe3340fe7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:33.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe3388c5700 1 --2- 172.21.15.88:0/1370891029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe334100100 0x7fe33419cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:33.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe3337fe700 1 -- 172.21.15.88:0/1370891029 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3340fed50 msgr2=0x7fe3340fe7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:33.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe3337fe700 1 --2- 172.21.15.88:0/1370891029 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3340fed50 0x7fe3340fe7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:33.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe3337fe700 1 -- 172.21.15.88:0/1370891029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe334100100 msgr2=0x7fe33419cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:33.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe3337fe700 1 --2- 172.21.15.88:0/1370891029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe334100100 0x7fe33419cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:33.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe3337fe700 1 -- 172.21.15.88:0/1370891029 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31c0095f0 con 0x7fe3340ff700 2023-07-14T17:42:33.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe3388c5700 1 --2- 172.21.15.88:0/1370891029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe334100100 0x7fe33419cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:42:33.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe333fff700 1 --2- 172.21.15.88:0/1370891029 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3340fed50 0x7fe3340fe7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:42:33.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe3337fe700 1 --2- 172.21.15.88:0/1370891029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3340ff700 0x7fe33419a520 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe31c006010 tx=0x7fe31c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:33.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe3317fa700 1 -- 172.21.15.88:0/1370891029 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c01d070 con 0x7fe3340ff700 2023-07-14T17:42:33.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.977+0000 7fe3317fa700 1 -- 172.21.15.88:0/1370891029 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe31c0041b0 con 0x7fe3340ff700 2023-07-14T17:42:33.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.978+0000 7fe33a328700 1 -- 172.21.15.88:0/1370891029 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe33419d8b0 con 0x7fe3340ff700 2023-07-14T17:42:33.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.978+0000 7fe3317fa700 1 -- 172.21.15.88:0/1370891029 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c022da0 con 0x7fe3340ff700 2023-07-14T17:42:33.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.978+0000 7fe33a328700 1 -- 172.21.15.88:0/1370891029 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe33419dcd0 con 0x7fe3340ff700 2023-07-14T17:42:33.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.978+0000 7fe33a328700 1 -- 172.21.15.88:0/1370891029 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe334106120 con 0x7fe3340ff700 2023-07-14T17:42:33.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.979+0000 7fe3317fa700 1 -- 172.21.15.88:0/1370891029 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe31c022570 con 0x7fe3340ff700 2023-07-14T17:42:33.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.980+0000 7fe3317fa700 1 --2- 172.21.15.88:0/1370891029 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe3200a11a0 0x7fe3200a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:33.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.980+0000 7fe333fff700 1 --2- 172.21.15.88:0/1370891029 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe3200a11a0 0x7fe3200a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:33.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.980+0000 7fe3317fa700 1 -- 172.21.15.88:0/1370891029 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe31c0cd9b0 con 0x7fe3340ff700 2023-07-14T17:42:33.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.981+0000 7fe333fff700 1 --2- 172.21.15.88:0/1370891029 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe3200a11a0 0x7fe3200a3650 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe324005fd0 tx=0x7fe324005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:33.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:33.984+0000 7fe3317fa700 1 -- 172.21.15.88:0/1370891029 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe31c097630 con 0x7fe3340ff700 2023-07-14T17:42:34.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:34.233+0000 7fe33a328700 1 -- 172.21.15.88:0/1370891029 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe33402c8e0 con 0x7fe3340ff700 2023-07-14T17:42:34.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:34.234+0000 7fe3317fa700 1 -- 172.21.15.88:0/1370891029 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8091 (secure 0 0 0) 0x7fe31c09ac50 con 0x7fe3340ff700 2023-07-14T17:42:34.217 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:34.218 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/3046592155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3046592155},{"type":"v1","addr":"172.21.15.88:6839","nonce":3046592155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/2332025068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":2332025068},{"type":"v1","addr":"172.21.15.178:6835","nonce":2332025068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3062637672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3062637672},{"type":"v1","addr":"172.21.15.107:6835","nonce":3062637672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/3950404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3950404084},{"type":"v1","addr":"172.21.15.178:6837","nonce":3950404084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24469,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/2953415077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2953415077},{"type":"v1","addr":"172.21.15.178:6839","nonce":2953415077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1152608615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1152608615},{"type":"v1","addr":"172.21.15.107:6837","nonce":1152608615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-07-14T17:42:34.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:34.236+0000 7fe33a328700 1 -- 172.21.15.88:0/1370891029 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe3200a11a0 msgr2=0x7fe3200a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:34.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:34.236+0000 7fe33a328700 1 --2- 172.21.15.88:0/1370891029 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe3200a11a0 0x7fe3200a3650 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe324005fd0 tx=0x7fe324005db0 comp rx=0 tx=0).stop 2023-07-14T17:42:34.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:34.236+0000 7fe33a328700 1 -- 172.21.15.88:0/1370891029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3340ff700 msgr2=0x7fe33419a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:34.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:34.236+0000 7fe33a328700 1 --2- 172.21.15.88:0/1370891029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3340ff700 0x7fe33419a520 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe31c006010 tx=0x7fe31c004730 comp rx=0 tx=0).stop 2023-07-14T17:42:34.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:34.237+0000 7fe33a328700 1 -- 172.21.15.88:0/1370891029 shutdown_connections 2023-07-14T17:42:34.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:34.237+0000 7fe33a328700 1 --2- 172.21.15.88:0/1370891029 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe334100100 0x7fe33419cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:34.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:34.237+0000 7fe33a328700 1 --2- 172.21.15.88:0/1370891029 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe3340fed50 0x7fe3340fe7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:34.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:34.237+0000 7fe33a328700 1 --2- 172.21.15.88:0/1370891029 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe3200a11a0 0x7fe3200a3650 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:34.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:34.237+0000 7fe33a328700 1 --2- 172.21.15.88:0/1370891029 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe3340ff700 0x7fe33419a520 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:34.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:34.237+0000 7fe33a328700 1 -- 172.21.15.88:0/1370891029 >> 172.21.15.88:0/1370891029 conn(0x7fe3340f98d0 msgr2=0x7fe334103260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:34.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:34.237+0000 7fe33a328700 1 -- 172.21.15.88:0/1370891029 shutdown_connections 2023-07-14T17:42:34.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:34.237+0000 7fe33a328700 1 -- 172.21.15.88:0/1370891029 wait complete. 2023-07-14T17:42:34.221 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 11 2023-07-14T17:42:34.664 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:34.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:34.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:34.665 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:34.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:34.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:34.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:34 smithi178 ceph-mon[129368]: Deploying daemon mds.a on smithi088 2023-07-14T17:42:34.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:34 smithi178 ceph-mon[129368]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:34.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:34 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.88:6838/3046592155,v1:172.21.15.88:6839/3046592155] up:boot 2023-07-14T17:42:34.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:34 smithi178 ceph-mon[129368]: fsmap 10 up:standby 2023-07-14T17:42:34.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:34 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T17:42:34.666 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:34 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1370891029' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:34.677 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:34.678 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:34.678 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:34.678 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:34.678 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:34.678 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:34.678 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:34 smithi088 ceph-mon[127465]: Deploying daemon mds.a on smithi088 2023-07-14T17:42:34.679 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:34 smithi088 ceph-mon[127465]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:34.679 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:34 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.88:6838/3046592155,v1:172.21.15.88:6839/3046592155] up:boot 2023-07-14T17:42:34.679 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:34 smithi088 ceph-mon[127465]: fsmap 10 up:standby 2023-07-14T17:42:34.679 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:34 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T17:42:34.679 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:34 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1370891029' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:34.724 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:34 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:34.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:34 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:34.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:34 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:34.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:34 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:34.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:34 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:34.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:34 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:34.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:34 smithi107 ceph-mon[128999]: Deploying daemon mds.a on smithi088 2023-07-14T17:42:34.725 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:34 smithi107 ceph-mon[128999]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:34.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:34 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.88:6838/3046592155,v1:172.21.15.88:6839/3046592155] up:boot 2023-07-14T17:42:34.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:34 smithi107 ceph-mon[128999]: fsmap 10 up:standby 2023-07-14T17:42:34.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:34 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T17:42:34.726 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:34 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1370891029' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:35.282 INFO:journalctl@ceph.mds.a.smithi088.stdout:Jul 14 17:42:35 smithi088 systemd[1]: Starting Ceph mds.a for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:42:35.560 INFO:journalctl@ceph.mds.a.smithi088.stdout:Jul 14 17:42:35 smithi088 podman[162099]: 2023-07-14T17:42:35.942 INFO:journalctl@ceph.mds.a.smithi088.stdout:Jul 14 17:42:35 smithi088 ceph-mds[162139]: -- 172.21.15.88:0/1113714209 <== mon.0 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557a44c331e0 con 0x557a44c08800 2023-07-14T17:42:35.942 INFO:journalctl@ceph.mds.a.smithi088.stdout:Jul 14 17:42:35 smithi088 bash[162099]: 987cb7698269deacdab65ab3474a79ba68cfa86423b2a4b1a040541d1985f715 2023-07-14T17:42:35.942 INFO:journalctl@ceph.mds.a.smithi088.stdout:Jul 14 17:42:35 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-a[162115]: starting mds.a at 2023-07-14T17:42:35.942 INFO:journalctl@ceph.mds.a.smithi088.stdout:Jul 14 17:42:35 smithi088 systemd[1]: Started Ceph mds.a for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:42:36.234 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:42:36.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.384+0000 7facec230700 1 -- 172.21.15.88:0/2474800245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face4100690 msgr2=0x7face4106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:36.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.384+0000 7facec230700 1 --2- 172.21.15.88:0/2474800245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face4100690 0x7face4106f50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7facdc00b0a0 tx=0x7facdc00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:36.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.384+0000 7facec230700 1 -- 172.21.15.88:0/2474800245 shutdown_connections 2023-07-14T17:42:36.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.384+0000 7facec230700 1 --2- 172.21.15.88:0/2474800245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face4100690 0x7face4106f50 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:36.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.384+0000 7facec230700 1 --2- 172.21.15.88:0/2474800245 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7face40feb10 0x7face40fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:36.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.384+0000 7facec230700 1 --2- 172.21.15.88:0/2474800245 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7face40ffd00 0x7face4100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:36.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.384+0000 7facec230700 1 -- 172.21.15.88:0/2474800245 >> 172.21.15.88:0/2474800245 conn(0x7face40fa130 msgr2=0x7face40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:36.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.384+0000 7facec230700 1 -- 172.21.15.88:0/2474800245 shutdown_connections 2023-07-14T17:42:36.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.384+0000 7facec230700 1 -- 172.21.15.88:0/2474800245 wait complete. 2023-07-14T17:42:36.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7facec230700 1 Processor -- start 2023-07-14T17:42:36.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7facec230700 1 -- start start 2023-07-14T17:42:36.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7facec230700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7face40feb10 0x7face4195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:36.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7facec230700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7face40ffd00 0x7face4195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:36.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7facec230700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face4100690 0x7face419cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:36.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7facec230700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face419d510 con 0x7face4100690 2023-07-14T17:42:36.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7facec230700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face419d650 con 0x7face40feb10 2023-07-14T17:42:36.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7facec230700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face419d7e0 con 0x7face40ffd00 2023-07-14T17:42:36.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7facea7cd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face4100690 0x7face419cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:36.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7facea7cd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face4100690 0x7face419cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41314/0 (socket says 172.21.15.88:41314) 2023-07-14T17:42:36.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7facea7cd700 1 -- 172.21.15.88:0/2581994567 learned_addr learned my addr 172.21.15.88:0/2581994567 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:36.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7face97cb700 1 --2- 172.21.15.88:0/2581994567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7face40ffd00 0x7face4195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:36.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7face9fcc700 1 --2- 172.21.15.88:0/2581994567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7face40feb10 0x7face4195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:36.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7facea7cd700 1 -- 172.21.15.88:0/2581994567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7face40ffd00 msgr2=0x7face4195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:36.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7facea7cd700 1 --2- 172.21.15.88:0/2581994567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7face40ffd00 0x7face4195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:36.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7facea7cd700 1 -- 172.21.15.88:0/2581994567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7face40feb10 msgr2=0x7face4195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:36.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7facea7cd700 1 --2- 172.21.15.88:0/2581994567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7face40feb10 0x7face4195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:36.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7facea7cd700 1 -- 172.21.15.88:0/2581994567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc009d00 con 0x7face4100690 2023-07-14T17:42:36.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.385+0000 7facea7cd700 1 --2- 172.21.15.88:0/2581994567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face4100690 0x7face419cf40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7facdc00b830 tx=0x7facdc004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:36.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.386+0000 7facdaffd700 1 -- 172.21.15.88:0/2581994567 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc00e040 con 0x7face4100690 2023-07-14T17:42:36.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.386+0000 7facdaffd700 1 -- 172.21.15.88:0/2581994567 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7facdc009440 con 0x7face4100690 2023-07-14T17:42:36.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.386+0000 7facec230700 1 -- 172.21.15.88:0/2581994567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face419da60 con 0x7face4100690 2023-07-14T17:42:36.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.386+0000 7facdaffd700 1 -- 172.21.15.88:0/2581994567 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc01db60 con 0x7face4100690 2023-07-14T17:42:36.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.386+0000 7facec230700 1 -- 172.21.15.88:0/2581994567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face419df00 con 0x7face4100690 2023-07-14T17:42:36.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.387+0000 7facec230700 1 -- 172.21.15.88:0/2581994567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face410a2a0 con 0x7face4100690 2023-07-14T17:42:36.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.388+0000 7facdaffd700 1 -- 172.21.15.88:0/2581994567 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7facdc005450 con 0x7face4100690 2023-07-14T17:42:36.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.388+0000 7facdaffd700 1 --2- 172.21.15.88:0/2581994567 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7facd00a11a0 0x7facd00a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:36.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.388+0000 7face9fcc700 1 --2- 172.21.15.88:0/2581994567 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7facd00a11a0 0x7facd00a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:36.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.388+0000 7facdaffd700 1 -- 172.21.15.88:0/2581994567 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7facdc0ce240 con 0x7face4100690 2023-07-14T17:42:36.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.389+0000 7face9fcc700 1 --2- 172.21.15.88:0/2581994567 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7facd00a11a0 0x7facd00a3650 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7face40ffb60 tx=0x7facd4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:36.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.392+0000 7facdaffd700 1 -- 172.21.15.88:0/2581994567 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7facdc097ec0 con 0x7face4100690 2023-07-14T17:42:36.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.621+0000 7facec230700 1 -- 172.21.15.88:0/2581994567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7face419e4b0 con 0x7face4100690 2023-07-14T17:42:36.605 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.622+0000 7facdaffd700 1 -- 172.21.15.88:0/2581994567 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8849 (secure 0 0 0) 0x7facdc005700 con 0x7face4100690 2023-07-14T17:42:36.605 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:36.606 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/3046592155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3046592155},{"type":"v1","addr":"172.21.15.88:6839","nonce":3046592155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6841/2509502918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6840","nonce":2509502918},{"type":"v1","addr":"172.21.15.88:6841","nonce":2509502918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/2332025068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":2332025068},{"type":"v1","addr":"172.21.15.178:6835","nonce":2332025068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3062637672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3062637672},{"type":"v1","addr":"172.21.15.107:6835","nonce":3062637672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/3950404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3950404084},{"type":"v1","addr":"172.21.15.178:6837","nonce":3950404084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24469,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/2953415077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2953415077},{"type":"v1","addr":"172.21.15.178:6839","nonce":2953415077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1152608615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1152608615},{"type":"v1","addr":"172.21.15.107:6837","nonce":1152608615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-07-14T17:42:36.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.624+0000 7facec230700 1 -- 172.21.15.88:0/2581994567 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7facd00a11a0 msgr2=0x7facd00a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:36.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.624+0000 7facec230700 1 --2- 172.21.15.88:0/2581994567 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7facd00a11a0 0x7facd00a3650 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7face40ffb60 tx=0x7facd4009190 comp rx=0 tx=0).stop 2023-07-14T17:42:36.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.624+0000 7facec230700 1 -- 172.21.15.88:0/2581994567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face4100690 msgr2=0x7face419cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:36.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.624+0000 7facec230700 1 --2- 172.21.15.88:0/2581994567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face4100690 0x7face419cf40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7facdc00b830 tx=0x7facdc004ce0 comp rx=0 tx=0).stop 2023-07-14T17:42:36.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.624+0000 7facec230700 1 -- 172.21.15.88:0/2581994567 shutdown_connections 2023-07-14T17:42:36.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.624+0000 7facec230700 1 --2- 172.21.15.88:0/2581994567 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7facd00a11a0 0x7facd00a3650 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:36.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.624+0000 7facec230700 1 --2- 172.21.15.88:0/2581994567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7face4100690 0x7face419cf40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:36.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.624+0000 7facec230700 1 --2- 172.21.15.88:0/2581994567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7face40feb10 0x7face4195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:36.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.624+0000 7facec230700 1 --2- 172.21.15.88:0/2581994567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7face40ffd00 0x7face4195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:36.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.624+0000 7facec230700 1 -- 172.21.15.88:0/2581994567 >> 172.21.15.88:0/2581994567 conn(0x7face40fa130 msgr2=0x7face4068a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:36.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.624+0000 7facec230700 1 -- 172.21.15.88:0/2581994567 shutdown_connections 2023-07-14T17:42:36.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:36.624+0000 7facec230700 1 -- 172.21.15.88:0/2581994567 wait complete. 2023-07-14T17:42:36.609 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 12 2023-07-14T17:42:36.852 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:36 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:36.853 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:36 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:36.853 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:36 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.88:6840/2509502918,v1:172.21.15.88:6841/2509502918] up:boot 2023-07-14T17:42:36.853 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:36 smithi107 ceph-mon[128999]: fsmap 11 up:standby 2023-07-14T17:42:36.853 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:36 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T17:42:36.853 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:36 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:36.854 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:36 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:36.854 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:36 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:36.854 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:36 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:36.854 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:36 smithi107 ceph-mon[128999]: Deploying daemon mds.b on smithi107 2023-07-14T17:42:36.854 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:36 smithi107 ceph-mon[128999]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:36.854 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:36 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2581994567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:36.914 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:36 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:36.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:36 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:36.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:36 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.88:6840/2509502918,v1:172.21.15.88:6841/2509502918] up:boot 2023-07-14T17:42:36.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:36 smithi178 ceph-mon[129368]: fsmap 11 up:standby 2023-07-14T17:42:36.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:36 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T17:42:36.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:36 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:36.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:36 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:36.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:36 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:36.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:36 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:36.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:36 smithi178 ceph-mon[129368]: Deploying daemon mds.b on smithi107 2023-07-14T17:42:36.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:36 smithi178 ceph-mon[129368]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:36.916 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:36 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2581994567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:36.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:36 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:36.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:36 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:36.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:36 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.88:6840/2509502918,v1:172.21.15.88:6841/2509502918] up:boot 2023-07-14T17:42:36.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:36 smithi088 ceph-mon[127465]: fsmap 11 up:standby 2023-07-14T17:42:36.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:36 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T17:42:36.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:36 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:36.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:36 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T17:42:36.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:36 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T17:42:36.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:36 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:36.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:36 smithi088 ceph-mon[127465]: Deploying daemon mds.b on smithi107 2023-07-14T17:42:36.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:36 smithi088 ceph-mon[127465]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:36.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:36 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2581994567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:37.475 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:42:37 smithi107 systemd[1]: Starting Ceph mds.b for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:42:37.808 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:42:37 smithi107 podman[154063]: 2023-07-14T17:42:38.224 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:42:37 smithi107 ceph-mds[154103]: -- 172.21.15.107:0/4052749560 <== mon.2 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559b8b56b1e0 con 0x559b8b541000 2023-07-14T17:42:38.225 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:42:37 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-b[154079]: starting mds.b at 2023-07-14T17:42:38.225 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:42:37 smithi107 bash[154063]: d13fd2934eaadc70e7c144fec90814a20ec0b508fd5189e589b98f5cd586697f 2023-07-14T17:42:38.225 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:42:37 smithi107 systemd[1]: Started Ceph mds.b for b462a556-226c-11ee-9b33-001a4aab830c. 2023-07-14T17:42:38.621 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:42:38.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.766+0000 7fa8d4390700 1 -- 172.21.15.88:0/689399692 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8cc1017b0 msgr2=0x7fa8cc108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:38.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.766+0000 7fa8d4390700 1 --2- 172.21.15.88:0/689399692 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8cc1017b0 0x7fa8cc108070 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa8c400b0a0 tx=0x7fa8c400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:38.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.766+0000 7fa8d4390700 1 -- 172.21.15.88:0/689399692 shutdown_connections 2023-07-14T17:42:38.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.766+0000 7fa8d4390700 1 --2- 172.21.15.88:0/689399692 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8cc1017b0 0x7fa8cc108070 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:38.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.766+0000 7fa8d4390700 1 --2- 172.21.15.88:0/689399692 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8cc0ffc30 0x7fa8cc100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:38.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.766+0000 7fa8d4390700 1 --2- 172.21.15.88:0/689399692 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8cc100e20 0x7fa8cc101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:38.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.766+0000 7fa8d4390700 1 -- 172.21.15.88:0/689399692 >> 172.21.15.88:0/689399692 conn(0x7fa8cc0fb230 msgr2=0x7fa8cc0fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:38.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.766+0000 7fa8d4390700 1 -- 172.21.15.88:0/689399692 shutdown_connections 2023-07-14T17:42:38.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.766+0000 7fa8d4390700 1 -- 172.21.15.88:0/689399692 wait complete. 2023-07-14T17:42:38.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d4390700 1 Processor -- start 2023-07-14T17:42:38.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d4390700 1 -- start start 2023-07-14T17:42:38.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d4390700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8cc0ffc30 0x7fa8cc0743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:38.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d4390700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8cc100e20 0x7fa8cc072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:38.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d4390700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8cc1017b0 0x7fa8cc072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:38.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d4390700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc073710 con 0x7fa8cc1017b0 2023-07-14T17:42:38.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d4390700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc073880 con 0x7fa8cc100e20 2023-07-14T17:42:38.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d4390700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc19fa20 con 0x7fa8cc0ffc30 2023-07-14T17:42:38.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d292d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8cc1017b0 0x7fa8cc072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:38.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d292d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8cc1017b0 0x7fa8cc072f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41326/0 (socket says 172.21.15.88:41326) 2023-07-14T17:42:38.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d292d700 1 -- 172.21.15.88:0/3996863803 learned_addr learned my addr 172.21.15.88:0/3996863803 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:38.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d192b700 1 --2- 172.21.15.88:0/3996863803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8cc100e20 0x7fa8cc072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:38.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d292d700 1 -- 172.21.15.88:0/3996863803 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8cc0ffc30 msgr2=0x7fa8cc0743f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:38.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d212c700 1 --2- 172.21.15.88:0/3996863803 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8cc0ffc30 0x7fa8cc0743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:38.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d292d700 1 --2- 172.21.15.88:0/3996863803 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8cc0ffc30 0x7fa8cc0743f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:38.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d292d700 1 -- 172.21.15.88:0/3996863803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8cc100e20 msgr2=0x7fa8cc072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:38.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d292d700 1 --2- 172.21.15.88:0/3996863803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8cc100e20 0x7fa8cc072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:38.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d292d700 1 -- 172.21.15.88:0/3996863803 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c4009d00 con 0x7fa8cc1017b0 2023-07-14T17:42:38.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d192b700 1 --2- 172.21.15.88:0/3996863803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8cc100e20 0x7fa8cc072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:42:38.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.767+0000 7fa8d212c700 1 --2- 172.21.15.88:0/3996863803 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8cc0ffc30 0x7fa8cc0743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:42:38.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.768+0000 7fa8d292d700 1 --2- 172.21.15.88:0/3996863803 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8cc1017b0 0x7fa8cc072f70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa8c400b830 tx=0x7fa8c4004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:38.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.768+0000 7fa8c37fe700 1 -- 172.21.15.88:0/3996863803 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c400e040 con 0x7fa8cc1017b0 2023-07-14T17:42:38.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.768+0000 7fa8d4390700 1 -- 172.21.15.88:0/3996863803 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8cc19fca0 con 0x7fa8cc1017b0 2023-07-14T17:42:38.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.768+0000 7fa8c37fe700 1 -- 172.21.15.88:0/3996863803 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa8c4009440 con 0x7fa8cc1017b0 2023-07-14T17:42:38.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.768+0000 7fa8c37fe700 1 -- 172.21.15.88:0/3996863803 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c401dbe0 con 0x7fa8cc1017b0 2023-07-14T17:42:38.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.768+0000 7fa8d4390700 1 -- 172.21.15.88:0/3996863803 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8cc1a0120 con 0x7fa8cc1017b0 2023-07-14T17:42:38.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.769+0000 7fa8d4390700 1 -- 172.21.15.88:0/3996863803 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8cc107110 con 0x7fa8cc1017b0 2023-07-14T17:42:38.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.770+0000 7fa8c37fe700 1 -- 172.21.15.88:0/3996863803 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa8c4005450 con 0x7fa8cc1017b0 2023-07-14T17:42:38.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.770+0000 7fa8c37fe700 1 --2- 172.21.15.88:0/3996863803 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa8b80a11a0 0x7fa8b80a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:38.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.770+0000 7fa8d212c700 1 --2- 172.21.15.88:0/3996863803 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa8b80a11a0 0x7fa8b80a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:38.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.770+0000 7fa8c37fe700 1 -- 172.21.15.88:0/3996863803 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa8c40ce200 con 0x7fa8cc1017b0 2023-07-14T17:42:38.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.771+0000 7fa8d212c700 1 --2- 172.21.15.88:0/3996863803 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa8b80a11a0 0x7fa8b80a3650 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc100c80 tx=0x7fa8c8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:38.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:38.774+0000 7fa8c37fe700 1 -- 172.21.15.88:0/3996863803 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa8c4097e80 con 0x7fa8cc1017b0 2023-07-14T17:42:38.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:38 smithi088 ceph-mon[127465]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:38.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:38 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] up:boot 2023-07-14T17:42:38.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:38 smithi088 ceph-mon[127465]: fsmap 12 up:standby 2023-07-14T17:42:38.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:38 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T17:42:38.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:38 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:38.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:38 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:38.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:38 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:38.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:38 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:38.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:38 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:38.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:38 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:42:38.943 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:38 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:39.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.021+0000 7fa8d4390700 1 -- 172.21.15.88:0/3996863803 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8cc02c8e0 con 0x7fa8cc1017b0 2023-07-14T17:42:39.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.022+0000 7fa8c37fe700 1 -- 172.21.15.88:0/3996863803 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9604 (secure 0 0 0) 0x7fa8c409b4a0 con 0x7fa8cc1017b0 2023-07-14T17:42:39.006 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:39.006 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/3046592155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3046592155},{"type":"v1","addr":"172.21.15.88:6839","nonce":3046592155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6841/2509502918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6840","nonce":2509502918},{"type":"v1","addr":"172.21.15.88:6841","nonce":2509502918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/2332025068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":2332025068},{"type":"v1","addr":"172.21.15.178:6835","nonce":2332025068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3062637672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3062637672},{"type":"v1","addr":"172.21.15.107:6835","nonce":3062637672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/3950404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3950404084},{"type":"v1","addr":"172.21.15.178:6837","nonce":3950404084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24469,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/2953415077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2953415077},{"type":"v1","addr":"172.21.15.178:6839","nonce":2953415077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1152608615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1152608615},{"type":"v1","addr":"172.21.15.107:6837","nonce":1152608615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24506,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6839/31646548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":31646548},{"type":"v1","addr":"172.21.15.107:6839","nonce":31646548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-07-14T17:42:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.024+0000 7fa8d4390700 1 -- 172.21.15.88:0/3996863803 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa8b80a11a0 msgr2=0x7fa8b80a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.024+0000 7fa8d4390700 1 --2- 172.21.15.88:0/3996863803 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa8b80a11a0 0x7fa8b80a3650 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc100c80 tx=0x7fa8c8008040 comp rx=0 tx=0).stop 2023-07-14T17:42:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.024+0000 7fa8d4390700 1 -- 172.21.15.88:0/3996863803 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8cc1017b0 msgr2=0x7fa8cc072f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.024+0000 7fa8d4390700 1 --2- 172.21.15.88:0/3996863803 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8cc1017b0 0x7fa8cc072f70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa8c400b830 tx=0x7fa8c4004d90 comp rx=0 tx=0).stop 2023-07-14T17:42:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.024+0000 7fa8d4390700 1 -- 172.21.15.88:0/3996863803 shutdown_connections 2023-07-14T17:42:39.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.024+0000 7fa8d4390700 1 --2- 172.21.15.88:0/3996863803 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa8b80a11a0 0x7fa8b80a3650 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.024+0000 7fa8d4390700 1 --2- 172.21.15.88:0/3996863803 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8cc1017b0 0x7fa8cc072f70 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.024+0000 7fa8d4390700 1 --2- 172.21.15.88:0/3996863803 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa8cc0ffc30 0x7fa8cc0743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.024+0000 7fa8d4390700 1 --2- 172.21.15.88:0/3996863803 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa8cc100e20 0x7fa8cc072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.024+0000 7fa8d4390700 1 -- 172.21.15.88:0/3996863803 >> 172.21.15.88:0/3996863803 conn(0x7fa8cc0fb230 msgr2=0x7fa8cc104250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.024+0000 7fa8d4390700 1 -- 172.21.15.88:0/3996863803 shutdown_connections 2023-07-14T17:42:39.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.024+0000 7fa8d4390700 1 -- 172.21.15.88:0/3996863803 wait complete. 2023-07-14T17:42:39.010 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 13 2023-07-14T17:42:39.020 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:42:39.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.163+0000 7f03abd87700 1 -- 172.21.15.88:0/1947068625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a4100680 msgr2=0x7f03a4106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:39.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.163+0000 7f03abd87700 1 --2- 172.21.15.88:0/1947068625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a4100680 0x7f03a4106f40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f039c00b0a0 tx=0x7f039c00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.164+0000 7f03abd87700 1 -- 172.21.15.88:0/1947068625 shutdown_connections 2023-07-14T17:42:39.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.164+0000 7f03abd87700 1 --2- 172.21.15.88:0/1947068625 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a4100680 0x7f03a4106f40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.164+0000 7f03abd87700 1 --2- 172.21.15.88:0/1947068625 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03a40feb00 0x7f03a40fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.164+0000 7f03abd87700 1 --2- 172.21.15.88:0/1947068625 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03a40ffcf0 0x7f03a4100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.164+0000 7f03abd87700 1 -- 172.21.15.88:0/1947068625 >> 172.21.15.88:0/1947068625 conn(0x7f03a40fa100 msgr2=0x7f03a40fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:39.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.164+0000 7f03abd87700 1 -- 172.21.15.88:0/1947068625 shutdown_connections 2023-07-14T17:42:39.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.164+0000 7f03abd87700 1 -- 172.21.15.88:0/1947068625 wait complete. 2023-07-14T17:42:39.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.164+0000 7f03abd87700 1 Processor -- start 2023-07-14T17:42:39.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.164+0000 7f03abd87700 1 -- start start 2023-07-14T17:42:39.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.164+0000 7f03abd87700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a40feb00 0x7f03a4068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:39.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.164+0000 7f03abd87700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03a40ffcf0 0x7f03a419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:39.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.164+0000 7f03abd87700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03a4100680 0x7f03a419ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:39.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.164+0000 7f03abd87700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a419d470 con 0x7f03a40feb00 2023-07-14T17:42:39.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.164+0000 7f03abd87700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a419d5e0 con 0x7f03a4100680 2023-07-14T17:42:39.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.164+0000 7f03abd87700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a419d7a0 con 0x7f03a40ffcf0 2023-07-14T17:42:39.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.164+0000 7f03a9b23700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a40feb00 0x7f03a4068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:39.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.165+0000 7f03a9b23700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a40feb00 0x7f03a4068370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41336/0 (socket says 172.21.15.88:41336) 2023-07-14T17:42:39.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.165+0000 7f03a9b23700 1 -- 172.21.15.88:0/2518809572 learned_addr learned my addr 172.21.15.88:0/2518809572 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:39.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.165+0000 7f03a9322700 1 --2- 172.21.15.88:0/2518809572 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03a40ffcf0 0x7f03a419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:39.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.165+0000 7f03aa324700 1 --2- 172.21.15.88:0/2518809572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03a4100680 0x7f03a419ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:39.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.165+0000 7f03a9b23700 1 -- 172.21.15.88:0/2518809572 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03a40ffcf0 msgr2=0x7f03a419a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:39.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.165+0000 7f03a9b23700 1 --2- 172.21.15.88:0/2518809572 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03a40ffcf0 0x7f03a419a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.165+0000 7f03a9b23700 1 -- 172.21.15.88:0/2518809572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03a4100680 msgr2=0x7f03a419ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:39.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.165+0000 7f03a9b23700 1 --2- 172.21.15.88:0/2518809572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03a4100680 0x7f03a419ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.165+0000 7f03a9b23700 1 -- 172.21.15.88:0/2518809572 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c009d00 con 0x7f03a40feb00 2023-07-14T17:42:39.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.165+0000 7f03aa324700 1 --2- 172.21.15.88:0/2518809572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03a4100680 0x7f03a419ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:42:39.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.165+0000 7f03a9b23700 1 --2- 172.21.15.88:0/2518809572 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a40feb00 0x7f03a4068370 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f03a000e970 tx=0x7f03a000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:39.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.165+0000 7f039affd700 1 -- 172.21.15.88:0/2518809572 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a0009790 con 0x7f03a40feb00 2023-07-14T17:42:39.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.165+0000 7f039affd700 1 -- 172.21.15.88:0/2518809572 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f03a000eec0 con 0x7f03a40feb00 2023-07-14T17:42:39.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.165+0000 7f03abd87700 1 -- 172.21.15.88:0/2518809572 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a419da80 con 0x7f03a40feb00 2023-07-14T17:42:39.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.165+0000 7f039affd700 1 -- 172.21.15.88:0/2518809572 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a0003c10 con 0x7f03a40feb00 2023-07-14T17:42:39.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.165+0000 7f03abd87700 1 -- 172.21.15.88:0/2518809572 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a419df90 con 0x7f03a40feb00 2023-07-14T17:42:39.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.166+0000 7f03abd87700 1 -- 172.21.15.88:0/2518809572 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a4105ff0 con 0x7f03a40feb00 2023-07-14T17:42:39.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.167+0000 7f039affd700 1 -- 172.21.15.88:0/2518809572 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f03a0003d90 con 0x7f03a40feb00 2023-07-14T17:42:39.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.168+0000 7f039affd700 1 --2- 172.21.15.88:0/2518809572 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f03900a11a0 0x7f03900a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:39.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.168+0000 7f03a9322700 1 --2- 172.21.15.88:0/2518809572 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f03900a11a0 0x7f03900a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:39.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.168+0000 7f039affd700 1 -- 172.21.15.88:0/2518809572 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f03a0014070 con 0x7f03a40feb00 2023-07-14T17:42:39.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.168+0000 7f03a9322700 1 --2- 172.21.15.88:0/2518809572 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f03900a11a0 0x7f03900a3650 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0394009c60 tx=0x7f0394009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:39.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.171+0000 7f039affd700 1 -- 172.21.15.88:0/2518809572 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f03a0096440 con 0x7f03a40feb00 2023-07-14T17:42:39.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:38 smithi178 ceph-mon[129368]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:39.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:38 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] up:boot 2023-07-14T17:42:39.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:38 smithi178 ceph-mon[129368]: fsmap 12 up:standby 2023-07-14T17:42:39.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:38 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T17:42:39.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:38 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:39.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:38 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:39.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:38 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:39.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:38 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:39.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:38 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:39.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:38 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:42:39.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:38 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:39.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:38 smithi107 ceph-mon[128999]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:39.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:38 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] up:boot 2023-07-14T17:42:39.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:38 smithi107 ceph-mon[128999]: fsmap 12 up:standby 2023-07-14T17:42:39.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:38 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T17:42:39.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:38 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:39.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:38 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:39.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:38 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:39.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:38 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:39.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:38 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:39.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:38 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:42:39.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:38 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:39.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.372+0000 7f03abd87700 1 -- 172.21.15.88:0/2518809572 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03a402c9d0 con 0x7f03a40feb00 2023-07-14T17:42:39.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.373+0000 7f039affd700 1 -- 172.21.15.88:0/2518809572 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+18447 (secure 0 0 0) 0x7f03a0099a60 con 0x7f03a40feb00 2023-07-14T17:42:39.357 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:39.357 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":78,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:42:02.830643+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:42:39.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.376+0000 7f03abd87700 1 -- 172.21.15.88:0/2518809572 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f03900a11a0 msgr2=0x7f03900a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:39.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.376+0000 7f03abd87700 1 --2- 172.21.15.88:0/2518809572 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f03900a11a0 0x7f03900a3650 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0394009c60 tx=0x7f0394009260 comp rx=0 tx=0).stop 2023-07-14T17:42:39.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.376+0000 7f03abd87700 1 -- 172.21.15.88:0/2518809572 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a40feb00 msgr2=0x7f03a4068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:39.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.376+0000 7f03abd87700 1 --2- 172.21.15.88:0/2518809572 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a40feb00 0x7f03a4068370 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f03a000e970 tx=0x7f03a000ec80 comp rx=0 tx=0).stop 2023-07-14T17:42:39.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.376+0000 7f03abd87700 1 -- 172.21.15.88:0/2518809572 shutdown_connections 2023-07-14T17:42:39.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.376+0000 7f03abd87700 1 --2- 172.21.15.88:0/2518809572 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f03a4100680 0x7f03a419ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.376+0000 7f03abd87700 1 --2- 172.21.15.88:0/2518809572 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f03900a11a0 0x7f03900a3650 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.376+0000 7f03abd87700 1 --2- 172.21.15.88:0/2518809572 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f03a40feb00 0x7f03a4068370 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.376+0000 7f03abd87700 1 --2- 172.21.15.88:0/2518809572 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f03a40ffcf0 0x7f03a419a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.376+0000 7f03abd87700 1 -- 172.21.15.88:0/2518809572 >> 172.21.15.88:0/2518809572 conn(0x7f03a40fa100 msgr2=0x7f03a4103130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:39.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.376+0000 7f03abd87700 1 -- 172.21.15.88:0/2518809572 shutdown_connections 2023-07-14T17:42:39.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.376+0000 7f03abd87700 1 -- 172.21.15.88:0/2518809572 wait complete. 2023-07-14T17:42:39.374 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T17:38:50.126535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T17:42:39.374 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T17:42:39.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.523+0000 7fab647bd700 1 -- 172.21.15.88:0/4242219034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab5c0fead0 msgr2=0x7fab5c0feee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:39.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.523+0000 7fab647bd700 1 --2- 172.21.15.88:0/4242219034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab5c0fead0 0x7fab5c0feee0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fab4c009990 tx=0x7fab4c009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.523+0000 7fab647bd700 1 -- 172.21.15.88:0/4242219034 shutdown_connections 2023-07-14T17:42:39.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.523+0000 7fab647bd700 1 --2- 172.21.15.88:0/4242219034 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab5c100650 0x7fab5c106f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.523+0000 7fab647bd700 1 --2- 172.21.15.88:0/4242219034 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab5c0fead0 0x7fab5c0feee0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.523+0000 7fab647bd700 1 --2- 172.21.15.88:0/4242219034 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab5c0ffcc0 0x7fab5c100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.523+0000 7fab647bd700 1 -- 172.21.15.88:0/4242219034 >> 172.21.15.88:0/4242219034 conn(0x7fab5c0fa110 msgr2=0x7fab5c0fc520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:39.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.523+0000 7fab647bd700 1 -- 172.21.15.88:0/4242219034 shutdown_connections 2023-07-14T17:42:39.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.523+0000 7fab647bd700 1 -- 172.21.15.88:0/4242219034 wait complete. 2023-07-14T17:42:39.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.524+0000 7fab647bd700 1 Processor -- start 2023-07-14T17:42:39.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.524+0000 7fab647bd700 1 -- start start 2023-07-14T17:42:39.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.524+0000 7fab647bd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab5c0fead0 0x7fab5c197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:39.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.524+0000 7fab647bd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab5c0ffcc0 0x7fab5c197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:39.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.524+0000 7fab647bd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab5c100650 0x7fab5c19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:39.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.524+0000 7fab647bd700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab5c19f5f0 con 0x7fab5c0ffcc0 2023-07-14T17:42:39.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.524+0000 7fab647bd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab5c19f730 con 0x7fab5c100650 2023-07-14T17:42:39.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.524+0000 7fab647bd700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab5c19f8c0 con 0x7fab5c0fead0 2023-07-14T17:42:39.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.524+0000 7fab61d58700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab5c0ffcc0 0x7fab5c197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:39.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.524+0000 7fab61d58700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab5c0ffcc0 0x7fab5c197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41350/0 (socket says 172.21.15.88:41350) 2023-07-14T17:42:39.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.524+0000 7fab61d58700 1 -- 172.21.15.88:0/4214715930 learned_addr learned my addr 172.21.15.88:0/4214715930 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:39.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.524+0000 7fab62d5a700 1 --2- 172.21.15.88:0/4214715930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab5c100650 0x7fab5c19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:39.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.524+0000 7fab62559700 1 --2- 172.21.15.88:0/4214715930 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab5c0fead0 0x7fab5c197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:39.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.525+0000 7fab61d58700 1 -- 172.21.15.88:0/4214715930 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab5c0fead0 msgr2=0x7fab5c197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:39.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.525+0000 7fab61d58700 1 --2- 172.21.15.88:0/4214715930 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab5c0fead0 0x7fab5c197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.525+0000 7fab61d58700 1 -- 172.21.15.88:0/4214715930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab5c100650 msgr2=0x7fab5c19f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:39.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.525+0000 7fab61d58700 1 --2- 172.21.15.88:0/4214715930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab5c100650 0x7fab5c19f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.525+0000 7fab61d58700 1 -- 172.21.15.88:0/4214715930 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4c0095f0 con 0x7fab5c0ffcc0 2023-07-14T17:42:39.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.525+0000 7fab61d58700 1 --2- 172.21.15.88:0/4214715930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab5c0ffcc0 0x7fab5c197fb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fab5800d980 tx=0x7fab5800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:39.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.525+0000 7fab537fe700 1 -- 172.21.15.88:0/4214715930 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58004220 con 0x7fab5c0ffcc0 2023-07-14T17:42:39.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.525+0000 7fab537fe700 1 -- 172.21.15.88:0/4214715930 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fab58007640 con 0x7fab5c0ffcc0 2023-07-14T17:42:39.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.525+0000 7fab647bd700 1 -- 172.21.15.88:0/4214715930 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab5c19fba0 con 0x7fab5c0ffcc0 2023-07-14T17:42:39.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.525+0000 7fab537fe700 1 -- 172.21.15.88:0/4214715930 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab580055e0 con 0x7fab5c0ffcc0 2023-07-14T17:42:39.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.525+0000 7fab647bd700 1 -- 172.21.15.88:0/4214715930 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab5c1a00b0 con 0x7fab5c0ffcc0 2023-07-14T17:42:39.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.526+0000 7fab647bd700 1 -- 172.21.15.88:0/4214715930 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab5c10a1a0 con 0x7fab5c0ffcc0 2023-07-14T17:42:39.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.527+0000 7fab537fe700 1 -- 172.21.15.88:0/4214715930 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fab58004750 con 0x7fab5c0ffcc0 2023-07-14T17:42:39.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.527+0000 7fab537fe700 1 --2- 172.21.15.88:0/4214715930 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fab480a1150 0x7fab480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:39.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.528+0000 7fab62559700 1 --2- 172.21.15.88:0/4214715930 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fab480a1150 0x7fab480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:39.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.528+0000 7fab537fe700 1 -- 172.21.15.88:0/4214715930 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fab580cc790 con 0x7fab5c0ffcc0 2023-07-14T17:42:39.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.528+0000 7fab62559700 1 --2- 172.21.15.88:0/4214715930 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fab480a1150 0x7fab480a3600 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fab4c00c010 tx=0x7fab4c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:39.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.531+0000 7fab537fe700 1 -- 172.21.15.88:0/4214715930 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fab58096380 con 0x7fab5c0ffcc0 2023-07-14T17:42:39.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.743+0000 7fab647bd700 1 -- 172.21.15.88:0/4214715930 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fab5c1a0660 con 0x7fab5c0ffcc0 2023-07-14T17:42:39.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.743+0000 7fab537fe700 1 -- 172.21.15.88:0/4214715930 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab58005740 con 0x7fab5c0ffcc0 2023-07-14T17:42:39.727 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-07-14T17:42:39.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.745+0000 7fab647bd700 1 -- 172.21.15.88:0/4214715930 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fab480a1150 msgr2=0x7fab480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:39.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.745+0000 7fab647bd700 1 --2- 172.21.15.88:0/4214715930 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fab480a1150 0x7fab480a3600 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fab4c00c010 tx=0x7fab4c00b540 comp rx=0 tx=0).stop 2023-07-14T17:42:39.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.745+0000 7fab647bd700 1 -- 172.21.15.88:0/4214715930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab5c0ffcc0 msgr2=0x7fab5c197fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:39.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.745+0000 7fab647bd700 1 --2- 172.21.15.88:0/4214715930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab5c0ffcc0 0x7fab5c197fb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fab5800d980 tx=0x7fab5800dc90 comp rx=0 tx=0).stop 2023-07-14T17:42:39.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.745+0000 7fab647bd700 1 -- 172.21.15.88:0/4214715930 shutdown_connections 2023-07-14T17:42:39.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.745+0000 7fab647bd700 1 --2- 172.21.15.88:0/4214715930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fab5c100650 0x7fab5c19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.745+0000 7fab647bd700 1 --2- 172.21.15.88:0/4214715930 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fab5c0fead0 0x7fab5c197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.745+0000 7fab647bd700 1 --2- 172.21.15.88:0/4214715930 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fab480a1150 0x7fab480a3600 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.745+0000 7fab647bd700 1 --2- 172.21.15.88:0/4214715930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fab5c0ffcc0 0x7fab5c197fb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.745+0000 7fab647bd700 1 -- 172.21.15.88:0/4214715930 >> 172.21.15.88:0/4214715930 conn(0x7fab5c0fa110 msgr2=0x7fab5c0685a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:39.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.745+0000 7fab647bd700 1 -- 172.21.15.88:0/4214715930 shutdown_connections 2023-07-14T17:42:39.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.745+0000 7fab647bd700 1 -- 172.21.15.88:0/4214715930 wait complete. 2023-07-14T17:42:39.741 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-07-14T17:42:39.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.883+0000 7ff19ee13700 1 -- 172.21.15.88:0/2832654526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1980feb00 msgr2=0x7ff1980fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:39.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.883+0000 7ff19ee13700 1 --2- 172.21.15.88:0/2832654526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1980feb00 0x7ff1980fef10 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff18c009990 tx=0x7ff18c009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.884+0000 7ff19ee13700 1 -- 172.21.15.88:0/2832654526 shutdown_connections 2023-07-14T17:42:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.884+0000 7ff19ee13700 1 --2- 172.21.15.88:0/2832654526 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff198100680 0x7ff198106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.884+0000 7ff19ee13700 1 --2- 172.21.15.88:0/2832654526 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1980feb00 0x7ff1980fef10 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.884+0000 7ff19ee13700 1 --2- 172.21.15.88:0/2832654526 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1980ffcf0 0x7ff198100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.884+0000 7ff19ee13700 1 -- 172.21.15.88:0/2832654526 >> 172.21.15.88:0/2832654526 conn(0x7ff1980fa100 msgr2=0x7ff1980fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:39.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.884+0000 7ff19ee13700 1 -- 172.21.15.88:0/2832654526 shutdown_connections 2023-07-14T17:42:39.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.884+0000 7ff19ee13700 1 -- 172.21.15.88:0/2832654526 wait complete. 2023-07-14T17:42:39.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.884+0000 7ff19ee13700 1 Processor -- start 2023-07-14T17:42:39.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.884+0000 7ff19ee13700 1 -- start start 2023-07-14T17:42:39.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.885+0000 7ff19ee13700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff1980feb00 0x7ff1981958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:39.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.885+0000 7ff19ee13700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1980ffcf0 0x7ff198195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:39.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.885+0000 7ff19ee13700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff198100680 0x7ff19819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:39.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.885+0000 7ff19ee13700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19819d450 con 0x7ff198100680 2023-07-14T17:42:39.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.885+0000 7ff19ee13700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19819d590 con 0x7ff1980ffcf0 2023-07-14T17:42:39.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.885+0000 7ff19ee13700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19819d720 con 0x7ff1980feb00 2023-07-14T17:42:39.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.885+0000 7ff19d3b0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff198100680 0x7ff19819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:39.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.885+0000 7ff19d3b0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff198100680 0x7ff19819ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41364/0 (socket says 172.21.15.88:41364) 2023-07-14T17:42:39.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.885+0000 7ff19d3b0700 1 -- 172.21.15.88:0/2939022738 learned_addr learned my addr 172.21.15.88:0/2939022738 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:39.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.885+0000 7ff197fff700 1 --2- 172.21.15.88:0/2939022738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1980ffcf0 0x7ff198195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:39.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.885+0000 7ff19cbaf700 1 --2- 172.21.15.88:0/2939022738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff1980feb00 0x7ff1981958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:39.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.885+0000 7ff19d3b0700 1 -- 172.21.15.88:0/2939022738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff1980feb00 msgr2=0x7ff1981958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:39.872 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.885+0000 7ff19d3b0700 1 --2- 172.21.15.88:0/2939022738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff1980feb00 0x7ff1981958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.885+0000 7ff19d3b0700 1 -- 172.21.15.88:0/2939022738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1980ffcf0 msgr2=0x7ff198195e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:39.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.885+0000 7ff19d3b0700 1 --2- 172.21.15.88:0/2939022738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1980ffcf0 0x7ff198195e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:39.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.885+0000 7ff19d3b0700 1 -- 172.21.15.88:0/2939022738 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c0095f0 con 0x7ff198100680 2023-07-14T17:42:39.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.885+0000 7ff19d3b0700 1 --2- 172.21.15.88:0/2939022738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff198100680 0x7ff19819ce80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff190007ee0 tx=0x7ff19000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:39.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.887+0000 7ff195ffb700 1 -- 172.21.15.88:0/2939022738 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff190017070 con 0x7ff198100680 2023-07-14T17:42:39.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.887+0000 7ff19ee13700 1 -- 172.21.15.88:0/2939022738 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19819da00 con 0x7ff198100680 2023-07-14T17:42:39.873 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.887+0000 7ff195ffb700 1 -- 172.21.15.88:0/2939022738 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff19000f040 con 0x7ff198100680 2023-07-14T17:42:39.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.887+0000 7ff195ffb700 1 -- 172.21.15.88:0/2939022738 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff190005570 con 0x7ff198100680 2023-07-14T17:42:39.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.887+0000 7ff19ee13700 1 -- 172.21.15.88:0/2939022738 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19819df40 con 0x7ff198100680 2023-07-14T17:42:39.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.888+0000 7ff19ee13700 1 -- 172.21.15.88:0/2939022738 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff19810a200 con 0x7ff198100680 2023-07-14T17:42:39.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.889+0000 7ff195ffb700 1 -- 172.21.15.88:0/2939022738 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff19000db20 con 0x7ff198100680 2023-07-14T17:42:39.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.889+0000 7ff195ffb700 1 --2- 172.21.15.88:0/2939022738 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff1880a11a0 0x7ff1880a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:39.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.889+0000 7ff195ffb700 1 -- 172.21.15.88:0/2939022738 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff1900cbbf0 con 0x7ff198100680 2023-07-14T17:42:39.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.889+0000 7ff19cbaf700 1 --2- 172.21.15.88:0/2939022738 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff1880a11a0 0x7ff1880a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:39.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.890+0000 7ff19cbaf700 1 --2- 172.21.15.88:0/2939022738 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff1880a11a0 0x7ff1880a3650 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff18c00b5c0 tx=0x7ff18c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:39.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:39.895+0000 7ff195ffb700 1 -- 172.21.15.88:0/2939022738 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff1900957e0 con 0x7ff198100680 2023-07-14T17:42:40.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.122+0000 7ff19ee13700 1 -- 172.21.15.88:0/2939022738 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff19819e550 con 0x7ff198100680 2023-07-14T17:42:40.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.124+0000 7ff195ffb700 1 -- 172.21.15.88:0/2939022738 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff190098e00 con 0x7ff198100680 2023-07-14T17:42:40.107 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:40.107 INFO:teuthology.orchestra.run.smithi088.stdout:[ 2023-07-14T17:42:40.107 INFO:teuthology.orchestra.run.smithi088.stdout: { 2023-07-14T17:42:40.108 INFO:teuthology.orchestra.run.smithi088.stdout: "poolnum": 1, 2023-07-14T17:42:40.108 INFO:teuthology.orchestra.run.smithi088.stdout: "poolname": ".mgr" 2023-07-14T17:42:40.108 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-07-14T17:42:40.108 INFO:teuthology.orchestra.run.smithi088.stdout:] 2023-07-14T17:42:40.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.126+0000 7ff19ee13700 1 -- 172.21.15.88:0/2939022738 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff1880a11a0 msgr2=0x7ff1880a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:40.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.126+0000 7ff19ee13700 1 --2- 172.21.15.88:0/2939022738 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff1880a11a0 0x7ff1880a3650 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff18c00b5c0 tx=0x7ff18c005960 comp rx=0 tx=0).stop 2023-07-14T17:42:40.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.126+0000 7ff19ee13700 1 -- 172.21.15.88:0/2939022738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff198100680 msgr2=0x7ff19819ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:40.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.126+0000 7ff19ee13700 1 --2- 172.21.15.88:0/2939022738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff198100680 0x7ff19819ce80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff190007ee0 tx=0x7ff19000d3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:40.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.126+0000 7ff19ee13700 1 -- 172.21.15.88:0/2939022738 shutdown_connections 2023-07-14T17:42:40.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.126+0000 7ff19ee13700 1 --2- 172.21.15.88:0/2939022738 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff1880a11a0 0x7ff1880a3650 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:40.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.126+0000 7ff19ee13700 1 --2- 172.21.15.88:0/2939022738 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff198100680 0x7ff19819ce80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:40.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.126+0000 7ff19ee13700 1 --2- 172.21.15.88:0/2939022738 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff1980feb00 0x7ff1981958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:40.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.126+0000 7ff19ee13700 1 --2- 172.21.15.88:0/2939022738 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1980ffcf0 0x7ff198195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:40.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.126+0000 7ff19ee13700 1 -- 172.21.15.88:0/2939022738 >> 172.21.15.88:0/2939022738 conn(0x7ff1980fa100 msgr2=0x7ff1980faa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:40.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.126+0000 7ff19ee13700 1 -- 172.21.15.88:0/2939022738 shutdown_connections 2023-07-14T17:42:40.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.126+0000 7ff19ee13700 1 -- 172.21.15.88:0/2939022738 wait complete. 2023-07-14T17:42:40.121 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:39 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3996863803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:40.121 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:39 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2518809572' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:40.122 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:39 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/4214715930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:42:40.122 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-07-14T17:42:40.123 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-07-14T17:42:40.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:39 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3996863803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:40.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:39 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2518809572' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:40.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:39 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/4214715930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:42:40.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:39 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3996863803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:40.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:39 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2518809572' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:40.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:39 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/4214715930' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:42:40.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.270+0000 7fd0df4c0700 1 -- 172.21.15.88:0/2018505972 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d80ffc10 msgr2=0x7fd0d8100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:40.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.270+0000 7fd0df4c0700 1 --2- 172.21.15.88:0/2018505972 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d80ffc10 0x7fd0d8100020 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4009990 tx=0x7fd0d4009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:40.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.270+0000 7fd0df4c0700 1 -- 172.21.15.88:0/2018505972 shutdown_connections 2023-07-14T17:42:40.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.270+0000 7fd0df4c0700 1 --2- 172.21.15.88:0/2018505972 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0d8101850 0x7fd0d81080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:40.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.270+0000 7fd0df4c0700 1 --2- 172.21.15.88:0/2018505972 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d80ffc10 0x7fd0d8100020 secure :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4009990 tx=0x7fd0d4009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:40.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.270+0000 7fd0df4c0700 1 --2- 172.21.15.88:0/2018505972 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0d8100ea0 0x7fd0d8101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:40.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.270+0000 7fd0df4c0700 1 -- 172.21.15.88:0/2018505972 >> 172.21.15.88:0/2018505972 conn(0x7fd0d80fb210 msgr2=0x7fd0d80fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:40.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.270+0000 7fd0df4c0700 1 -- 172.21.15.88:0/2018505972 shutdown_connections 2023-07-14T17:42:40.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.270+0000 7fd0df4c0700 1 -- 172.21.15.88:0/2018505972 wait complete. 2023-07-14T17:42:40.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.270+0000 7fd0df4c0700 1 Processor -- start 2023-07-14T17:42:40.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0df4c0700 1 -- start start 2023-07-14T17:42:40.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0df4c0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d8100ea0 0x7fd0d8195ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:40.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0df4c0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0d8101850 0x7fd0d8196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:40.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0df4c0700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0d819b080 0x7fd0d819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:40.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0df4c0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d819da30 con 0x7fd0d8100ea0 2023-07-14T17:42:40.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0df4c0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d819db70 con 0x7fd0d8101850 2023-07-14T17:42:40.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0df4c0700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d819dd00 con 0x7fd0d819b080 2023-07-14T17:42:40.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0dd25c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d8100ea0 0x7fd0d8195ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:40.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0dd25c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d8100ea0 0x7fd0d8195ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41374/0 (socket says 172.21.15.88:41374) 2023-07-14T17:42:40.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0dd25c700 1 -- 172.21.15.88:0/2273847503 learned_addr learned my addr 172.21.15.88:0/2273847503 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:40.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0dda5d700 1 --2- 172.21.15.88:0/2273847503 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0d819b080 0x7fd0d819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:40.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0dca5b700 1 --2- 172.21.15.88:0/2273847503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0d8101850 0x7fd0d8196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:40.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0dd25c700 1 -- 172.21.15.88:0/2273847503 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0d819b080 msgr2=0x7fd0d819d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:40.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0dd25c700 1 --2- 172.21.15.88:0/2273847503 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0d819b080 0x7fd0d819d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:40.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0dd25c700 1 -- 172.21.15.88:0/2273847503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0d8101850 msgr2=0x7fd0d8196020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:40.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0dd25c700 1 --2- 172.21.15.88:0/2273847503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0d8101850 0x7fd0d8196020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:40.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0dd25c700 1 -- 172.21.15.88:0/2273847503 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d40095f0 con 0x7fd0d8100ea0 2023-07-14T17:42:40.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0dd25c700 1 --2- 172.21.15.88:0/2273847503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d8100ea0 0x7fd0d8195ae0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4005910 tx=0x7fd0d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:40.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.271+0000 7fd0ce7fc700 1 -- 172.21.15.88:0/2273847503 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d401d070 con 0x7fd0d8100ea0 2023-07-14T17:42:40.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.272+0000 7fd0df4c0700 1 -- 172.21.15.88:0/2273847503 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d819df80 con 0x7fd0d8100ea0 2023-07-14T17:42:40.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.272+0000 7fd0ce7fc700 1 -- 172.21.15.88:0/2273847503 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd0d4005a00 con 0x7fd0d8100ea0 2023-07-14T17:42:40.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.272+0000 7fd0ce7fc700 1 -- 172.21.15.88:0/2273847503 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d400fcd0 con 0x7fd0d8100ea0 2023-07-14T17:42:40.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.272+0000 7fd0df4c0700 1 -- 172.21.15.88:0/2273847503 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d819e2f0 con 0x7fd0d8100ea0 2023-07-14T17:42:40.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.274+0000 7fd0ce7fc700 1 -- 172.21.15.88:0/2273847503 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd0d400f460 con 0x7fd0d8100ea0 2023-07-14T17:42:40.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.274+0000 7fd0df4c0700 1 -- 172.21.15.88:0/2273847503 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d810b290 con 0x7fd0d8100ea0 2023-07-14T17:42:40.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.275+0000 7fd0ce7fc700 1 --2- 172.21.15.88:0/2273847503 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd0c40a1150 0x7fd0c40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:40.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.275+0000 7fd0ce7fc700 1 -- 172.21.15.88:0/2273847503 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd0d40cd410 con 0x7fd0d8100ea0 2023-07-14T17:42:40.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.275+0000 7fd0dca5b700 1 --2- 172.21.15.88:0/2273847503 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd0c40a1150 0x7fd0c40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:40.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.276+0000 7fd0dca5b700 1 --2- 172.21.15.88:0/2273847503 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd0c40a1150 0x7fd0c40a3600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8009c60 tx=0x7fd0c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:40.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.280+0000 7fd0ce7fc700 1 -- 172.21.15.88:0/2273847503 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd0d4097090 con 0x7fd0d8100ea0 2023-07-14T17:42:40.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.476+0000 7fd0df4c0700 1 -- 172.21.15.88:0/2273847503 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd0d810b450 con 0x7fd0d8100ea0 2023-07-14T17:42:40.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.901+0000 7fd0ce7fc700 1 -- 172.21.15.88:0/2273847503 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v79) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd0d409a6b0 con 0x7fd0d8100ea0 2023-07-14T17:42:40.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.904+0000 7fd0df4c0700 1 -- 172.21.15.88:0/2273847503 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd0c40a1150 msgr2=0x7fd0c40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:40.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.904+0000 7fd0df4c0700 1 --2- 172.21.15.88:0/2273847503 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd0c40a1150 0x7fd0c40a3600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd0c8009c60 tx=0x7fd0c8009260 comp rx=0 tx=0).stop 2023-07-14T17:42:40.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.904+0000 7fd0df4c0700 1 -- 172.21.15.88:0/2273847503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d8100ea0 msgr2=0x7fd0d8195ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:40.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.904+0000 7fd0df4c0700 1 --2- 172.21.15.88:0/2273847503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d8100ea0 0x7fd0d8195ae0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4005910 tx=0x7fd0d4004730 comp rx=0 tx=0).stop 2023-07-14T17:42:40.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.905+0000 7fd0df4c0700 1 -- 172.21.15.88:0/2273847503 shutdown_connections 2023-07-14T17:42:40.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.905+0000 7fd0df4c0700 1 --2- 172.21.15.88:0/2273847503 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0d819b080 0x7fd0d819d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:40.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.905+0000 7fd0df4c0700 1 --2- 172.21.15.88:0/2273847503 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd0c40a1150 0x7fd0c40a3600 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:40.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.905+0000 7fd0df4c0700 1 --2- 172.21.15.88:0/2273847503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0d8100ea0 0x7fd0d8195ae0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:40.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.905+0000 7fd0df4c0700 1 --2- 172.21.15.88:0/2273847503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd0d8101850 0x7fd0d8196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:40.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.905+0000 7fd0df4c0700 1 -- 172.21.15.88:0/2273847503 >> 172.21.15.88:0/2273847503 conn(0x7fd0d80fb210 msgr2=0x7fd0d80fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:40.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.905+0000 7fd0df4c0700 1 -- 172.21.15.88:0/2273847503 shutdown_connections 2023-07-14T17:42:40.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:40.905+0000 7fd0df4c0700 1 -- 172.21.15.88:0/2273847503 wait complete. 2023-07-14T17:42:40.890 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' created 2023-07-14T17:42:40.901 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-07-14T17:42:41.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.060+0000 7f1c73e8e700 1 -- 172.21.15.88:0/1897373251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c6c0ff7b0 msgr2=0x7f1c6c0ffc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:41.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.060+0000 7f1c73e8e700 1 --2- 172.21.15.88:0/1897373251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c6c0ff7b0 0x7f1c6c0ffc20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1c6400b0a0 tx=0x7f1c6400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:41.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.060+0000 7f1c73e8e700 1 -- 172.21.15.88:0/1897373251 shutdown_connections 2023-07-14T17:42:41.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.060+0000 7f1c73e8e700 1 --2- 172.21.15.88:0/1897373251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c6c0ff7b0 0x7f1c6c0ffc20 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:41.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.060+0000 7f1c73e8e700 1 --2- 172.21.15.88:0/1897373251 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c0fe4b0 0x7f1c6c0fe8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:41.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.060+0000 7f1c73e8e700 1 --2- 172.21.15.88:0/1897373251 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c6c0fee00 0x7f1c6c0ff270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:41.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.060+0000 7f1c73e8e700 1 -- 172.21.15.88:0/1897373251 >> 172.21.15.88:0/1897373251 conn(0x7f1c6c0fa160 msgr2=0x7f1c6c0fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:41.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.060+0000 7f1c73e8e700 1 -- 172.21.15.88:0/1897373251 shutdown_connections 2023-07-14T17:42:41.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.060+0000 7f1c73e8e700 1 -- 172.21.15.88:0/1897373251 wait complete. 2023-07-14T17:42:41.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c73e8e700 1 Processor -- start 2023-07-14T17:42:41.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c73e8e700 1 -- start start 2023-07-14T17:42:41.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c73e8e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c0fe4b0 0x7f1c6c197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:41.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c73e8e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c6c0fee00 0x7f1c6c1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:41.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c73e8e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c6c0ff7b0 0x7f1c6c19f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:41.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c73e8e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c19f6e0 con 0x7f1c6c0ff7b0 2023-07-14T17:42:41.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c73e8e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c19f820 con 0x7f1c6c0fe4b0 2023-07-14T17:42:41.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c73e8e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c19f9b0 con 0x7f1c6c0fee00 2023-07-14T17:42:41.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c7242b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c6c0ff7b0 0x7f1c6c19f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:41.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c7242b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c6c0ff7b0 0x7f1c6c19f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41390/0 (socket says 172.21.15.88:41390) 2023-07-14T17:42:41.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c71c2a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c0fe4b0 0x7f1c6c197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:41.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c7242b700 1 -- 172.21.15.88:0/2817526074 learned_addr learned my addr 172.21.15.88:0/2817526074 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:41.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c71429700 1 --2- 172.21.15.88:0/2817526074 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c6c0fee00 0x7f1c6c1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:41.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c7242b700 1 -- 172.21.15.88:0/2817526074 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c6c0fee00 msgr2=0x7f1c6c1980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:41.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c7242b700 1 --2- 172.21.15.88:0/2817526074 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c6c0fee00 0x7f1c6c1980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:41.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c7242b700 1 -- 172.21.15.88:0/2817526074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c0fe4b0 msgr2=0x7f1c6c197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:41.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c7242b700 1 --2- 172.21.15.88:0/2817526074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c0fe4b0 0x7f1c6c197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:41.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c7242b700 1 -- 172.21.15.88:0/2817526074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c68009520 con 0x7f1c6c0ff7b0 2023-07-14T17:42:41.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.061+0000 7f1c71c2a700 1 --2- 172.21.15.88:0/2817526074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c0fe4b0 0x7f1c6c197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T17:42:41.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.062+0000 7f1c7242b700 1 --2- 172.21.15.88:0/2817526074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c6c0ff7b0 0x7f1c6c19f110 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1c6400b780 tx=0x7f1c640078e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:41.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.062+0000 7f1c62ffd700 1 -- 172.21.15.88:0/2817526074 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6400e040 con 0x7f1c6c0ff7b0 2023-07-14T17:42:41.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.062+0000 7f1c62ffd700 1 -- 172.21.15.88:0/2817526074 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1c6400be80 con 0x7f1c6c0ff7b0 2023-07-14T17:42:41.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.062+0000 7f1c62ffd700 1 -- 172.21.15.88:0/2817526074 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6401dea0 con 0x7f1c6c0ff7b0 2023-07-14T17:42:41.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.062+0000 7f1c73e8e700 1 -- 172.21.15.88:0/2817526074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c64009d00 con 0x7f1c6c0ff7b0 2023-07-14T17:42:41.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.062+0000 7f1c73e8e700 1 -- 172.21.15.88:0/2817526074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c6c19fed0 con 0x7f1c6c0ff7b0 2023-07-14T17:42:41.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.063+0000 7f1c73e8e700 1 -- 172.21.15.88:0/2817526074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c6c109290 con 0x7f1c6c0ff7b0 2023-07-14T17:42:41.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.064+0000 7f1c62ffd700 1 -- 172.21.15.88:0/2817526074 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1c6401d470 con 0x7f1c6c0ff7b0 2023-07-14T17:42:41.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.064+0000 7f1c62ffd700 1 --2- 172.21.15.88:0/2817526074 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1c580a11d0 0x7f1c580a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:41.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.065+0000 7f1c71c2a700 1 --2- 172.21.15.88:0/2817526074 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1c580a11d0 0x7f1c580a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:41.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.065+0000 7f1c62ffd700 1 -- 172.21.15.88:0/2817526074 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f1c640cdb70 con 0x7f1c6c0ff7b0 2023-07-14T17:42:41.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.065+0000 7f1c71c2a700 1 --2- 172.21.15.88:0/2817526074 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1c580a11d0 0x7f1c580a3680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1c68009fd0 tx=0x7f1c68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:41.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.069+0000 7f1c62ffd700 1 -- 172.21.15.88:0/2817526074 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1c64097680 con 0x7f1c6c0ff7b0 2023-07-14T17:42:41.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:40 smithi178 ceph-mon[129368]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:41.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:40 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2939022738' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T17:42:41.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:40 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2273847503' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T17:42:41.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:40 smithi088 ceph-mon[127465]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:41.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:40 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2939022738' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T17:42:41.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:40 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2273847503' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T17:42:41.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:40 smithi107 ceph-mon[128999]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:41.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:40 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2939022738' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T17:42:41.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:40 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2273847503' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T17:42:41.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.264+0000 7f1c73e8e700 1 -- 172.21.15.88:0/2817526074 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f1c6c02c8e0 con 0x7f1c6c0ff7b0 2023-07-14T17:42:41.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.901+0000 7f1c62ffd700 1 -- 172.21.15.88:0/2817526074 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v80) v1 ==== 171+0+0 (secure 0 0 0) 0x7f1c6409aca0 con 0x7f1c6c0ff7b0 2023-07-14T17:42:41.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.911+0000 7f1c73e8e700 1 -- 172.21.15.88:0/2817526074 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1c580a11d0 msgr2=0x7f1c580a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:41.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.911+0000 7f1c73e8e700 1 --2- 172.21.15.88:0/2817526074 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1c580a11d0 0x7f1c580a3680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1c68009fd0 tx=0x7f1c68009260 comp rx=0 tx=0).stop 2023-07-14T17:42:41.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.911+0000 7f1c73e8e700 1 -- 172.21.15.88:0/2817526074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c6c0ff7b0 msgr2=0x7f1c6c19f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:41.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.911+0000 7f1c73e8e700 1 --2- 172.21.15.88:0/2817526074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c6c0ff7b0 0x7f1c6c19f110 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1c6400b780 tx=0x7f1c640078e0 comp rx=0 tx=0).stop 2023-07-14T17:42:41.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.912+0000 7f1c73e8e700 1 -- 172.21.15.88:0/2817526074 shutdown_connections 2023-07-14T17:42:41.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.912+0000 7f1c73e8e700 1 --2- 172.21.15.88:0/2817526074 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1c580a11d0 0x7f1c580a3680 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1c68009fd0 tx=0x7f1c68009260 comp rx=0 tx=0).stop 2023-07-14T17:42:41.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.912+0000 7f1c73e8e700 1 --2- 172.21.15.88:0/2817526074 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1c6c0ff7b0 0x7f1c6c19f110 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:41.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.912+0000 7f1c73e8e700 1 --2- 172.21.15.88:0/2817526074 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1c6c0fe4b0 0x7f1c6c197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:41.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.912+0000 7f1c73e8e700 1 --2- 172.21.15.88:0/2817526074 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1c6c0fee00 0x7f1c6c1980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:41.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.912+0000 7f1c73e8e700 1 -- 172.21.15.88:0/2817526074 >> 172.21.15.88:0/2817526074 conn(0x7f1c6c0fa160 msgr2=0x7f1c6c0fbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:41.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.913+0000 7f1c73e8e700 1 -- 172.21.15.88:0/2817526074 shutdown_connections 2023-07-14T17:42:41.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:41.915+0000 7f1c73e8e700 1 -- 172.21.15.88:0/2817526074 wait complete. 2023-07-14T17:42:41.903 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' created 2023-07-14T17:42:41.914 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-07-14T17:42:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.058+0000 7f70eab8f700 1 -- 172.21.15.88:0/2633139832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70e4100700 msgr2=0x7f70e4102ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.058+0000 7f70eab8f700 1 --2- 172.21.15.88:0/2633139832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70e4100700 0x7f70e4102ae0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f70d8009990 tx=0x7f70d8009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.058+0000 7f70eab8f700 1 -- 172.21.15.88:0/2633139832 shutdown_connections 2023-07-14T17:42:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.058+0000 7f70eab8f700 1 --2- 172.21.15.88:0/2633139832 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70e4105940 0x7f70e4107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:42.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.058+0000 7f70eab8f700 1 --2- 172.21.15.88:0/2633139832 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70e4100700 0x7f70e4102ae0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.058+0000 7f70eab8f700 1 --2- 172.21.15.88:0/2633139832 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e4103020 0x7f70e4105400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.058+0000 7f70eab8f700 1 -- 172.21.15.88:0/2633139832 >> 172.21.15.88:0/2633139832 conn(0x7f70e40fa160 msgr2=0x7f70e40fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.058+0000 7f70eab8f700 1 -- 172.21.15.88:0/2633139832 shutdown_connections 2023-07-14T17:42:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.058+0000 7f70eab8f700 1 -- 172.21.15.88:0/2633139832 wait complete. 2023-07-14T17:42:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70eab8f700 1 Processor -- start 2023-07-14T17:42:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70eab8f700 1 -- start start 2023-07-14T17:42:42.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70eab8f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e4100700 0x7f70e41958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70eab8f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70e4103020 0x7f70e4195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70eab8f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70e4105940 0x7f70e419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70eab8f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e419d450 con 0x7f70e4103020 2023-07-14T17:42:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70eab8f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e419d590 con 0x7f70e4100700 2023-07-14T17:42:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70eab8f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e419d720 con 0x7f70e4105940 2023-07-14T17:42:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70e3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70e4103020 0x7f70e4195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70e3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70e4103020 0x7f70e4195e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41400/0 (socket says 172.21.15.88:41400) 2023-07-14T17:42:42.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70e3fff700 1 -- 172.21.15.88:0/3980787020 learned_addr learned my addr 172.21.15.88:0/3980787020 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70e912c700 1 --2- 172.21.15.88:0/3980787020 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70e4105940 0x7f70e419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70e892b700 1 --2- 172.21.15.88:0/3980787020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e4100700 0x7f70e41958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70e3fff700 1 -- 172.21.15.88:0/3980787020 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70e4105940 msgr2=0x7f70e419ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70e3fff700 1 --2- 172.21.15.88:0/3980787020 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70e4105940 0x7f70e419ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70e3fff700 1 -- 172.21.15.88:0/3980787020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e4100700 msgr2=0x7f70e41958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70e3fff700 1 --2- 172.21.15.88:0/3980787020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e4100700 0x7f70e41958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:42.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70e3fff700 1 -- 172.21.15.88:0/3980787020 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d80095f0 con 0x7f70e4103020 2023-07-14T17:42:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.059+0000 7f70e3fff700 1 --2- 172.21.15.88:0/3980787020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70e4103020 0x7f70e4195e10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f70d000c920 tx=0x7f70d000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.060+0000 7f70e1ffb700 1 -- 172.21.15.88:0/3980787020 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d0007b00 con 0x7f70e4103020 2023-07-14T17:42:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.060+0000 7f70e1ffb700 1 -- 172.21.15.88:0/3980787020 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f70d000ce70 con 0x7f70e4103020 2023-07-14T17:42:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.060+0000 7f70e1ffb700 1 -- 172.21.15.88:0/3980787020 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d0005440 con 0x7f70e4103020 2023-07-14T17:42:42.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.060+0000 7f70eab8f700 1 -- 172.21.15.88:0/3980787020 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e419da00 con 0x7f70e4103020 2023-07-14T17:42:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.060+0000 7f70eab8f700 1 -- 172.21.15.88:0/3980787020 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e419dec0 con 0x7f70e4103020 2023-07-14T17:42:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.061+0000 7f70eab8f700 1 -- 172.21.15.88:0/3980787020 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e410b2d0 con 0x7f70e4103020 2023-07-14T17:42:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.062+0000 7f70e1ffb700 1 -- 172.21.15.88:0/3980787020 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f70d00055a0 con 0x7f70e4103020 2023-07-14T17:42:42.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.062+0000 7f70e1ffb700 1 --2- 172.21.15.88:0/3980787020 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f70d40a1150 0x7f70d40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.062+0000 7f70e892b700 1 --2- 172.21.15.88:0/3980787020 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f70d40a1150 0x7f70d40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.062+0000 7f70e1ffb700 1 -- 172.21.15.88:0/3980787020 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f70d00cc330 con 0x7f70e4103020 2023-07-14T17:42:42.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.063+0000 7f70e892b700 1 --2- 172.21.15.88:0/3980787020 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f70d40a1150 0x7f70d40a3600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f70d800b5c0 tx=0x7f70d8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:42.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.067+0000 7f70e1ffb700 1 -- 172.21.15.88:0/3980787020 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f70d0095c30 con 0x7f70e4103020 2023-07-14T17:42:42.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:41 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2273847503' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T17:42:42.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:41 smithi178 ceph-mon[129368]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T17:42:42.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:41 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:41 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:41 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:41 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:41 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:41 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:41 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2817526074' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-14T17:42:42.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:41 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:42.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:41 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:42:42.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:41 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:41 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:42:42.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:41 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2273847503' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T17:42:42.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:41 smithi088 ceph-mon[127465]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T17:42:42.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:41 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:41 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:41 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:41 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:41 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:41 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:41 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2817526074' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-14T17:42:42.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:41 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:42.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:41 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:42:42.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:41 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:41 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:42:42.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:41 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2273847503' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T17:42:42.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:41 smithi107 ceph-mon[128999]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T17:42:42.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:41 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:41 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:41 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:41 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:41 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:41 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:41 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2817526074' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-14T17:42:42.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:41 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:42.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:41 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:42:42.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:41 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:42.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:41 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:42:42.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.312+0000 7f70eab8f700 1 -- 172.21.15.88:0/3980787020 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f70e419e680 con 0x7f70e4103020 2023-07-14T17:42:42.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.913+0000 7f70e1ffb700 1 -- 172.21.15.88:0/3980787020 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f70d0099250 con 0x7f70e4103020 2023-07-14T17:42:42.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.915+0000 7f70eab8f700 1 -- 172.21.15.88:0/3980787020 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f70d40a1150 msgr2=0x7f70d40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:42.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.915+0000 7f70eab8f700 1 --2- 172.21.15.88:0/3980787020 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f70d40a1150 0x7f70d40a3600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f70d800b5c0 tx=0x7f70d8005fb0 comp rx=0 tx=0).stop 2023-07-14T17:42:42.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.915+0000 7f70eab8f700 1 -- 172.21.15.88:0/3980787020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70e4103020 msgr2=0x7f70e4195e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:42.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.915+0000 7f70eab8f700 1 --2- 172.21.15.88:0/3980787020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70e4103020 0x7f70e4195e10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f70d000c920 tx=0x7f70d000cc30 comp rx=0 tx=0).stop 2023-07-14T17:42:42.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.915+0000 7f70eab8f700 1 -- 172.21.15.88:0/3980787020 shutdown_connections 2023-07-14T17:42:42.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.915+0000 7f70eab8f700 1 --2- 172.21.15.88:0/3980787020 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f70e4105940 0x7f70e419ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:42.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.915+0000 7f70eab8f700 1 --2- 172.21.15.88:0/3980787020 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f70e4100700 0x7f70e41958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:42.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.915+0000 7f70eab8f700 1 --2- 172.21.15.88:0/3980787020 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f70d40a1150 0x7f70d40a3600 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:42.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.915+0000 7f70eab8f700 1 --2- 172.21.15.88:0/3980787020 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70e4103020 0x7f70e4195e10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:42.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.915+0000 7f70eab8f700 1 -- 172.21.15.88:0/3980787020 >> 172.21.15.88:0/3980787020 conn(0x7f70e40fa160 msgr2=0x7f70e4101b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:42.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.916+0000 7f70eab8f700 1 -- 172.21.15.88:0/3980787020 shutdown_connections 2023-07-14T17:42:42.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:42.916+0000 7f70eab8f700 1 -- 172.21.15.88:0/3980787020 wait complete. 2023-07-14T17:42:42.901 INFO:teuthology.orchestra.run.smithi088.stderr:new fs with metadata pool 2 and data pool 3 2023-07-14T17:42:42.913 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-07-14T17:42:42.914 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-07-14T17:42:43.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.065+0000 7fef0e133700 1 -- 172.21.15.88:0/1070460930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef080ffc10 msgr2=0x7fef08100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:43.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.065+0000 7fef0e133700 1 --2- 172.21.15.88:0/1070460930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef080ffc10 0x7fef08100020 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7feef8009990 tx=0x7feef8009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:43.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.065+0000 7fef0e133700 1 -- 172.21.15.88:0/1070460930 shutdown_connections 2023-07-14T17:42:43.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.065+0000 7fef0e133700 1 --2- 172.21.15.88:0/1070460930 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef08101790 0x7fef08108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:43.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.065+0000 7fef0e133700 1 --2- 172.21.15.88:0/1070460930 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef080ffc10 0x7fef08100020 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:43.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.065+0000 7fef0e133700 1 --2- 172.21.15.88:0/1070460930 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef08100e00 0x7fef08101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:43.049 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.065+0000 7fef0e133700 1 -- 172.21.15.88:0/1070460930 >> 172.21.15.88:0/1070460930 conn(0x7fef080fb250 msgr2=0x7fef080fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:43.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.065+0000 7fef0e133700 1 -- 172.21.15.88:0/1070460930 shutdown_connections 2023-07-14T17:42:43.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.065+0000 7fef0e133700 1 -- 172.21.15.88:0/1070460930 wait complete. 2023-07-14T17:42:43.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef0e133700 1 Processor -- start 2023-07-14T17:42:43.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef0e133700 1 -- start start 2023-07-14T17:42:43.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef0e133700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef080ffc10 0x7fef08195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:43.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef0e133700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef08100e00 0x7fef08195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:43.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef0e133700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef08101790 0x7fef0819ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:43.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef0e133700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0819d3e0 con 0x7fef08101790 2023-07-14T17:42:43.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef0e133700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0819d520 con 0x7fef08100e00 2023-07-14T17:42:43.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef0e133700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0819d6b0 con 0x7fef080ffc10 2023-07-14T17:42:43.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef07fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef08101790 0x7fef0819ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:43.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef07fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef08101790 0x7fef0819ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41424/0 (socket says 172.21.15.88:41424) 2023-07-14T17:42:43.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef07fff700 1 -- 172.21.15.88:0/1332744799 learned_addr learned my addr 172.21.15.88:0/1332744799 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:43.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef077fe700 1 --2- 172.21.15.88:0/1332744799 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef080ffc10 0x7fef08195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:43.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef06ffd700 1 --2- 172.21.15.88:0/1332744799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef08100e00 0x7fef08195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:43.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef07fff700 1 -- 172.21.15.88:0/1332744799 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef080ffc10 msgr2=0x7fef08195860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:43.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef07fff700 1 --2- 172.21.15.88:0/1332744799 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef080ffc10 0x7fef08195860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:43.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef07fff700 1 -- 172.21.15.88:0/1332744799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef08100e00 msgr2=0x7fef08195da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:43.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef07fff700 1 --2- 172.21.15.88:0/1332744799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef08100e00 0x7fef08195da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:43.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef07fff700 1 -- 172.21.15.88:0/1332744799 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef80095f0 con 0x7fef08101790 2023-07-14T17:42:43.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef06ffd700 1 --2- 172.21.15.88:0/1332744799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef08100e00 0x7fef08195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:42:43.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.066+0000 7fef07fff700 1 --2- 172.21.15.88:0/1332744799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef08101790 0x7fef0819ce10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fef0000ef30 tx=0x7fef0000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:43.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.067+0000 7fef04ff9700 1 -- 172.21.15.88:0/1332744799 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef00018070 con 0x7fef08101790 2023-07-14T17:42:43.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.067+0000 7fef0e133700 1 -- 172.21.15.88:0/1332744799 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef0819d930 con 0x7fef08101790 2023-07-14T17:42:43.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.067+0000 7fef04ff9700 1 -- 172.21.15.88:0/1332744799 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fef0000f040 con 0x7fef08101790 2023-07-14T17:42:43.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.067+0000 7fef04ff9700 1 -- 172.21.15.88:0/1332744799 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0000cc70 con 0x7fef08101790 2023-07-14T17:42:43.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.067+0000 7fef0e133700 1 -- 172.21.15.88:0/1332744799 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef0819de40 con 0x7fef08101790 2023-07-14T17:42:43.055 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.068+0000 7fef0e133700 1 -- 172.21.15.88:0/1332744799 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef0810b290 con 0x7fef08101790 2023-07-14T17:42:43.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.069+0000 7fef04ff9700 1 -- 172.21.15.88:0/1332744799 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fef00003a00 con 0x7fef08101790 2023-07-14T17:42:43.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.069+0000 7fef04ff9700 1 --2- 172.21.15.88:0/1332744799 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7feef00a1150 0x7feef00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:43.056 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.069+0000 7fef077fe700 1 --2- 172.21.15.88:0/1332744799 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7feef00a1150 0x7feef00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:43.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.069+0000 7fef04ff9700 1 -- 172.21.15.88:0/1332744799 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fef000cc900 con 0x7fef08101790 2023-07-14T17:42:43.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.070+0000 7fef077fe700 1 --2- 172.21.15.88:0/1332744799 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7feef00a1150 0x7feef00a3600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feef8006010 tx=0x7feef800b2a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:43.057 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.073+0000 7fef04ff9700 1 -- 172.21.15.88:0/1332744799 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fef00096230 con 0x7fef08101790 2023-07-14T17:42:43.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:42 smithi178 ceph-mon[129368]: pgmap v248: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:43.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:42 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2817526074' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-07-14T17:42:43.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:42 smithi178 ceph-mon[129368]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T17:42:43.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:42 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3980787020' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T17:42:43.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:42 smithi178 ceph-mon[129368]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T17:42:43.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:42 smithi178 ceph-mon[129368]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T17:42:43.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:42 smithi088 ceph-mon[127465]: pgmap v248: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:43.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:42 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2817526074' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-07-14T17:42:43.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:42 smithi088 ceph-mon[127465]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T17:42:43.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:42 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3980787020' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T17:42:43.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:42 smithi088 ceph-mon[127465]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T17:42:43.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:42 smithi088 ceph-mon[127465]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T17:42:43.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:42 smithi107 ceph-mon[128999]: pgmap v248: 33 pgs: 3 creating+peering, 29 unknown, 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T17:42:43.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:42 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2817526074' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-07-14T17:42:43.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:42 smithi107 ceph-mon[128999]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T17:42:43.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:42 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3980787020' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T17:42:43.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:42 smithi107 ceph-mon[128999]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T17:42:43.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:42 smithi107 ceph-mon[128999]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T17:42:43.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.279+0000 7fef0e133700 1 -- 172.21.15.88:0/1332744799 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fef08106bb0 con 0x7fef08101790 2023-07-14T17:42:43.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.523+0000 7fef04ff9700 1 -- 172.21.15.88:0/1332744799 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v82) v1 ==== 157+0+0 (secure 0 0 0) 0x7fef00003cb0 con 0x7fef08101790 2023-07-14T17:42:43.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.525+0000 7fef0e133700 1 -- 172.21.15.88:0/1332744799 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7feef00a1150 msgr2=0x7feef00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:43.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.525+0000 7fef0e133700 1 --2- 172.21.15.88:0/1332744799 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7feef00a1150 0x7feef00a3600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feef8006010 tx=0x7feef800b2a0 comp rx=0 tx=0).stop 2023-07-14T17:42:43.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.525+0000 7fef0e133700 1 -- 172.21.15.88:0/1332744799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef08101790 msgr2=0x7fef0819ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:43.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.525+0000 7fef0e133700 1 --2- 172.21.15.88:0/1332744799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef08101790 0x7fef0819ce10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fef0000ef30 tx=0x7fef0000c3a0 comp rx=0 tx=0).stop 2023-07-14T17:42:43.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.525+0000 7fef0e133700 1 -- 172.21.15.88:0/1332744799 shutdown_connections 2023-07-14T17:42:43.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.525+0000 7fef0e133700 1 --2- 172.21.15.88:0/1332744799 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7feef00a1150 0x7feef00a3600 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:43.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.525+0000 7fef0e133700 1 --2- 172.21.15.88:0/1332744799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef08101790 0x7fef0819ce10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:43.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.525+0000 7fef0e133700 1 --2- 172.21.15.88:0/1332744799 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef080ffc10 0x7fef08195860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:43.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.525+0000 7fef0e133700 1 --2- 172.21.15.88:0/1332744799 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef08100e00 0x7fef08195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:43.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.525+0000 7fef0e133700 1 -- 172.21.15.88:0/1332744799 >> 172.21.15.88:0/1332744799 conn(0x7fef080fb250 msgr2=0x7fef080fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:43.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.525+0000 7fef0e133700 1 -- 172.21.15.88:0/1332744799 shutdown_connections 2023-07-14T17:42:43.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.525+0000 7fef0e133700 1 -- 172.21.15.88:0/1332744799 wait complete. 2023-07-14T17:42:43.520 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-07-14T17:42:43.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.672+0000 7f827f116700 1 -- 172.21.15.88:0/501482821 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82780ffd00 msgr2=0x7f8278100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:43.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.672+0000 7f827f116700 1 --2- 172.21.15.88:0/501482821 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82780ffd00 0x7f8278100150 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8264009990 tx=0x7f8264009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:43.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.672+0000 7f827f116700 1 -- 172.21.15.88:0/501482821 shutdown_connections 2023-07-14T17:42:43.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.672+0000 7f827f116700 1 --2- 172.21.15.88:0/501482821 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8278100690 0x7f8278106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:43.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.672+0000 7f827f116700 1 --2- 172.21.15.88:0/501482821 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f82780feb10 0x7f82780fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:43.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.672+0000 7f827f116700 1 --2- 172.21.15.88:0/501482821 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82780ffd00 0x7f8278100150 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:43.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.672+0000 7f827f116700 1 -- 172.21.15.88:0/501482821 >> 172.21.15.88:0/501482821 conn(0x7f82780fa130 msgr2=0x7f82780fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:43.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.672+0000 7f827f116700 1 -- 172.21.15.88:0/501482821 shutdown_connections 2023-07-14T17:42:43.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.672+0000 7f827f116700 1 -- 172.21.15.88:0/501482821 wait complete. 2023-07-14T17:42:43.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.672+0000 7f827f116700 1 Processor -- start 2023-07-14T17:42:43.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.672+0000 7f827f116700 1 -- start start 2023-07-14T17:42:43.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.672+0000 7f827f116700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82780feb10 0x7f82781957c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:43.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.672+0000 7f827f116700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82780ffd00 0x7f8278195d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:43.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.672+0000 7f827f116700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8278100690 0x7f827819cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:43.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.672+0000 7f827f116700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827819d340 con 0x7f82780ffd00 2023-07-14T17:42:43.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.672+0000 7f827f116700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827819d480 con 0x7f82780feb10 2023-07-14T17:42:43.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.672+0000 7f827f116700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827819d610 con 0x7f8278100690 2023-07-14T17:42:43.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.673+0000 7f8277fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82780ffd00 0x7f8278195d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:43.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.673+0000 7f8277fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82780ffd00 0x7f8278195d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41436/0 (socket says 172.21.15.88:41436) 2023-07-14T17:42:43.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.673+0000 7f8277fff700 1 -- 172.21.15.88:0/1533539528 learned_addr learned my addr 172.21.15.88:0/1533539528 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:43.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.673+0000 7f827d6b3700 1 --2- 172.21.15.88:0/1533539528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8278100690 0x7f827819cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:43.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.673+0000 7f827ceb2700 1 --2- 172.21.15.88:0/1533539528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82780feb10 0x7f82781957c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:43.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.673+0000 7f8277fff700 1 -- 172.21.15.88:0/1533539528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8278100690 msgr2=0x7f827819cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:43.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.673+0000 7f8277fff700 1 --2- 172.21.15.88:0/1533539528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8278100690 0x7f827819cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:43.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.673+0000 7f8277fff700 1 -- 172.21.15.88:0/1533539528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82780feb10 msgr2=0x7f82781957c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:43.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.673+0000 7f8277fff700 1 --2- 172.21.15.88:0/1533539528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82780feb10 0x7f82781957c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:43.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.673+0000 7f8277fff700 1 -- 172.21.15.88:0/1533539528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82640095f0 con 0x7f82780ffd00 2023-07-14T17:42:43.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.673+0000 7f827d6b3700 1 --2- 172.21.15.88:0/1533539528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8278100690 0x7f827819cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:42:43.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.673+0000 7f8277fff700 1 --2- 172.21.15.88:0/1533539528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82780ffd00 0x7f8278195d00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8264003ec0 tx=0x7f8264004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:43.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.673+0000 7f8275ffb700 1 -- 172.21.15.88:0/1533539528 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826401d070 con 0x7f82780ffd00 2023-07-14T17:42:43.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.673+0000 7f8275ffb700 1 -- 172.21.15.88:0/1533539528 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f826400be90 con 0x7f82780ffd00 2023-07-14T17:42:43.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.673+0000 7f827f116700 1 -- 172.21.15.88:0/1533539528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827819d890 con 0x7f82780ffd00 2023-07-14T17:42:43.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.673+0000 7f8275ffb700 1 -- 172.21.15.88:0/1533539528 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264022e80 con 0x7f82780ffd00 2023-07-14T17:42:43.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.673+0000 7f827f116700 1 -- 172.21.15.88:0/1533539528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f827819dcb0 con 0x7f82780ffd00 2023-07-14T17:42:43.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.674+0000 7f827f116700 1 -- 172.21.15.88:0/1533539528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82781059f0 con 0x7f82780ffd00 2023-07-14T17:42:43.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.675+0000 7f8275ffb700 1 -- 172.21.15.88:0/1533539528 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8264022650 con 0x7f82780ffd00 2023-07-14T17:42:43.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.676+0000 7f8275ffb700 1 --2- 172.21.15.88:0/1533539528 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f82680a11f0 0x7f82680a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:43.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.676+0000 7f827ceb2700 1 --2- 172.21.15.88:0/1533539528 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f82680a11f0 0x7f82680a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:43.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.676+0000 7f8275ffb700 1 -- 172.21.15.88:0/1533539528 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8205+0+0 (secure 0 0 0) 0x7f82640ce260 con 0x7f82780ffd00 2023-07-14T17:42:43.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.676+0000 7f827ceb2700 1 --2- 172.21.15.88:0/1533539528 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f82680a11f0 0x7f82680a36a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f82780ffb60 tx=0x7f826c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:43.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.679+0000 7f8275ffb700 1 -- 172.21.15.88:0/1533539528 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8264097a60 con 0x7f82780ffd00 2023-07-14T17:42:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:43.885+0000 7f827f116700 1 -- 172.21.15.88:0/1533539528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f827802cb60 con 0x7f82780ffd00 2023-07-14T17:42:44.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:43 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3980787020' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T17:42:44.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:43 smithi178 ceph-mon[129368]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T17:42:44.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:43 smithi178 ceph-mon[129368]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T17:42:44.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:43 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1332744799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-14T17:42:44.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:43 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1332744799' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-07-14T17:42:44.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:43 smithi178 ceph-mon[129368]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T17:42:44.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:43 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1533539528' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T17:42:44.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:43 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3980787020' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T17:42:44.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:43 smithi088 ceph-mon[127465]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T17:42:44.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:43 smithi088 ceph-mon[127465]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T17:42:44.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:43 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1332744799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-14T17:42:44.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:43 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1332744799' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-07-14T17:42:44.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:43 smithi088 ceph-mon[127465]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T17:42:44.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:43 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1533539528' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T17:42:44.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:43 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3980787020' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T17:42:44.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:43 smithi107 ceph-mon[128999]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T17:42:44.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:43 smithi107 ceph-mon[128999]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T17:42:44.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:43 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1332744799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-14T17:42:44.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:43 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1332744799' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-07-14T17:42:44.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:43 smithi107 ceph-mon[128999]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T17:42:44.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:43 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1533539528' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T17:42:45.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:44 smithi178 ceph-mon[129368]: pgmap v252: 97 pgs: 3 creating+peering, 48 unknown, 46 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 6 op/s 2023-07-14T17:42:45.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:44 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] up:active 2023-07-14T17:42:45.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:44 smithi178 ceph-mon[129368]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T17:42:45.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:44 smithi178 ceph-mon[129368]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T17:42:45.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:45.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:42:45.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:44 smithi178 ceph-mon[129368]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T17:42:45.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:44 smithi088 ceph-mon[127465]: pgmap v252: 97 pgs: 3 creating+peering, 48 unknown, 46 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 6 op/s 2023-07-14T17:42:45.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:44 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] up:active 2023-07-14T17:42:45.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:44 smithi088 ceph-mon[127465]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T17:42:45.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:44 smithi088 ceph-mon[127465]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T17:42:45.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:45.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:42:45.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.194 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:44 smithi088 ceph-mon[127465]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T17:42:45.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:44 smithi107 ceph-mon[128999]: pgmap v252: 97 pgs: 3 creating+peering, 48 unknown, 46 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 6 op/s 2023-07-14T17:42:45.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:44 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] up:active 2023-07-14T17:42:45.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:44 smithi107 ceph-mon[128999]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T17:42:45.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:44 smithi107 ceph-mon[128999]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T17:42:45.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:42:45.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:42:45.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:42:45.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:44 smithi107 ceph-mon[128999]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T17:42:45.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.528+0000 7f8275ffb700 1 -- 172.21.15.88:0/1533539528 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v84) v1 ==== 228+0+0 (secure 0 0 0) 0x7f826409b080 con 0x7f82780ffd00 2023-07-14T17:42:45.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.533+0000 7f827f116700 1 -- 172.21.15.88:0/1533539528 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f82680a11f0 msgr2=0x7f82680a36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:45.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.533+0000 7f827f116700 1 --2- 172.21.15.88:0/1533539528 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f82680a11f0 0x7f82680a36a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f82780ffb60 tx=0x7f826c009260 comp rx=0 tx=0).stop 2023-07-14T17:42:45.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.533+0000 7f827f116700 1 -- 172.21.15.88:0/1533539528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82780ffd00 msgr2=0x7f8278195d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:45.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.533+0000 7f827f116700 1 --2- 172.21.15.88:0/1533539528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82780ffd00 0x7f8278195d00 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8264003ec0 tx=0x7f8264004730 comp rx=0 tx=0).stop 2023-07-14T17:42:45.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.534+0000 7f827f116700 1 -- 172.21.15.88:0/1533539528 shutdown_connections 2023-07-14T17:42:45.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.534+0000 7f827f116700 1 --2- 172.21.15.88:0/1533539528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8278100690 0x7f827819cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:45.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.534+0000 7f827f116700 1 --2- 172.21.15.88:0/1533539528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82780feb10 0x7f82781957c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:45.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.534+0000 7f827f116700 1 --2- 172.21.15.88:0/1533539528 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f82680a11f0 0x7f82680a36a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:45.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.534+0000 7f827f116700 1 --2- 172.21.15.88:0/1533539528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82780ffd00 0x7f8278195d00 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:45.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.534+0000 7f827f116700 1 -- 172.21.15.88:0/1533539528 >> 172.21.15.88:0/1533539528 conn(0x7f82780fa130 msgr2=0x7f82780fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:45.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.534+0000 7f827f116700 1 -- 172.21.15.88:0/1533539528 shutdown_connections 2023-07-14T17:42:45.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.535+0000 7f827f116700 1 -- 172.21.15.88:0/1533539528 wait complete. 2023-07-14T17:42:45.519 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data_ec' created 2023-07-14T17:42:45.533 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-07-14T17:42:45.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.680+0000 7f9d3fa9b700 1 -- 172.21.15.88:0/77633627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38100690 msgr2=0x7f9d38106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.680+0000 7f9d3fa9b700 1 --2- 172.21.15.88:0/77633627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38100690 0x7f9d38106f50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000b0a0 tx=0x7f9d3000b3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.681+0000 7f9d3fa9b700 1 -- 172.21.15.88:0/77633627 shutdown_connections 2023-07-14T17:42:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.681+0000 7f9d3fa9b700 1 --2- 172.21.15.88:0/77633627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38100690 0x7f9d38106f50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.681+0000 7f9d3fa9b700 1 --2- 172.21.15.88:0/77633627 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d380feb10 0x7f9d380fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.681+0000 7f9d3fa9b700 1 --2- 172.21.15.88:0/77633627 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d380ffd00 0x7f9d38100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:45.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.681+0000 7f9d3fa9b700 1 -- 172.21.15.88:0/77633627 >> 172.21.15.88:0/77633627 conn(0x7f9d380fa130 msgr2=0x7f9d380fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.681+0000 7f9d3fa9b700 1 -- 172.21.15.88:0/77633627 shutdown_connections 2023-07-14T17:42:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.681+0000 7f9d3fa9b700 1 -- 172.21.15.88:0/77633627 wait complete. 2023-07-14T17:42:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.681+0000 7f9d3fa9b700 1 Processor -- start 2023-07-14T17:42:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.681+0000 7f9d3fa9b700 1 -- start start 2023-07-14T17:42:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.681+0000 7f9d3fa9b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d380feb10 0x7f9d38068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:45.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.681+0000 7f9d3fa9b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d380ffd00 0x7f9d3819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.681+0000 7f9d3fa9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38100690 0x7f9d3819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.681+0000 7f9d3fa9b700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3819d4d0 con 0x7f9d38100690 2023-07-14T17:42:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.681+0000 7f9d3fa9b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3819d640 con 0x7f9d380feb10 2023-07-14T17:42:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.681+0000 7f9d3fa9b700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3819d800 con 0x7f9d380ffd00 2023-07-14T17:42:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.681+0000 7f9d3e038700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38100690 0x7f9d3819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.681+0000 7f9d3e038700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38100690 0x7f9d3819cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41456/0 (socket says 172.21.15.88:41456) 2023-07-14T17:42:45.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.682+0000 7f9d3e038700 1 -- 172.21.15.88:0/694153911 learned_addr learned my addr 172.21.15.88:0/694153911 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:45.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.682+0000 7f9d3d837700 1 --2- 172.21.15.88:0/694153911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d380feb10 0x7f9d38068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:45.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.682+0000 7f9d3d036700 1 --2- 172.21.15.88:0/694153911 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d380ffd00 0x7f9d3819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:45.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.682+0000 7f9d3e038700 1 -- 172.21.15.88:0/694153911 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d380ffd00 msgr2=0x7f9d3819a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:45.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.682+0000 7f9d3e038700 1 --2- 172.21.15.88:0/694153911 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d380ffd00 0x7f9d3819a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:45.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.682+0000 7f9d3e038700 1 -- 172.21.15.88:0/694153911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d380feb10 msgr2=0x7f9d38068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.682+0000 7f9d3e038700 1 --2- 172.21.15.88:0/694153911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d380feb10 0x7f9d38068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.682+0000 7f9d3e038700 1 -- 172.21.15.88:0/694153911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d30009d00 con 0x7f9d38100690 2023-07-14T17:42:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.682+0000 7f9d3e038700 1 --2- 172.21.15.88:0/694153911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38100690 0x7f9d3819cd30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000b070 tx=0x7f9d30004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.682+0000 7f9d2effd700 1 -- 172.21.15.88:0/694153911 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3000e040 con 0x7f9d38100690 2023-07-14T17:42:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.682+0000 7f9d2effd700 1 -- 172.21.15.88:0/694153911 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9d30003e90 con 0x7f9d38100690 2023-07-14T17:42:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.682+0000 7f9d3fa9b700 1 -- 172.21.15.88:0/694153911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d3819da80 con 0x7f9d38100690 2023-07-14T17:42:45.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.682+0000 7f9d2effd700 1 -- 172.21.15.88:0/694153911 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3001b450 con 0x7f9d38100690 2023-07-14T17:42:45.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.682+0000 7f9d3fa9b700 1 -- 172.21.15.88:0/694153911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d3819df50 con 0x7f9d38100690 2023-07-14T17:42:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.683+0000 7f9d3fa9b700 1 -- 172.21.15.88:0/694153911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d38105a40 con 0x7f9d38100690 2023-07-14T17:42:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.684+0000 7f9d2effd700 1 -- 172.21.15.88:0/694153911 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9d30004000 con 0x7f9d38100690 2023-07-14T17:42:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.685+0000 7f9d2effd700 1 --2- 172.21.15.88:0/694153911 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9d240a11d0 0x7f9d240a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.685+0000 7f9d3d837700 1 --2- 172.21.15.88:0/694153911 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9d240a11d0 0x7f9d240a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.685+0000 7f9d2effd700 1 -- 172.21.15.88:0/694153911 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f9d300ce650 con 0x7f9d38100690 2023-07-14T17:42:45.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.685+0000 7f9d3d837700 1 --2- 172.21.15.88:0/694153911 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9d240a11d0 0x7f9d240a3680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9d380ffb60 tx=0x7f9d3400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:45.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.688+0000 7f9d2effd700 1 -- 172.21.15.88:0/694153911 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9d30097c60 con 0x7f9d38100690 2023-07-14T17:42:45.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:45.876+0000 7f9d3fa9b700 1 -- 172.21.15.88:0/694153911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f9d3802cb50 con 0x7f9d38100690 2023-07-14T17:42:46.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.527+0000 7f9d2effd700 1 -- 172.21.15.88:0/694153911 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v85) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9d30011390 con 0x7f9d38100690 2023-07-14T17:42:46.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.537+0000 7f9d3fa9b700 1 -- 172.21.15.88:0/694153911 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9d240a11d0 msgr2=0x7f9d240a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:46.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.537+0000 7f9d3fa9b700 1 --2- 172.21.15.88:0/694153911 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9d240a11d0 0x7f9d240a3680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9d380ffb60 tx=0x7f9d3400b410 comp rx=0 tx=0).stop 2023-07-14T17:42:46.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.537+0000 7f9d3fa9b700 1 -- 172.21.15.88:0/694153911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38100690 msgr2=0x7f9d3819cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:46.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.537+0000 7f9d3fa9b700 1 --2- 172.21.15.88:0/694153911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38100690 0x7f9d3819cd30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9d3000b070 tx=0x7f9d30004c80 comp rx=0 tx=0).stop 2023-07-14T17:42:46.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.537+0000 7f9d3fa9b700 1 -- 172.21.15.88:0/694153911 shutdown_connections 2023-07-14T17:42:46.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.537+0000 7f9d3fa9b700 1 --2- 172.21.15.88:0/694153911 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9d240a11d0 0x7f9d240a3680 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:46.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.537+0000 7f9d3fa9b700 1 --2- 172.21.15.88:0/694153911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d38100690 0x7f9d3819cd30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:46.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.537+0000 7f9d3fa9b700 1 --2- 172.21.15.88:0/694153911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9d380feb10 0x7f9d38068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:46.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.537+0000 7f9d3fa9b700 1 --2- 172.21.15.88:0/694153911 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9d380ffd00 0x7f9d3819a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:46.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.537+0000 7f9d3fa9b700 1 -- 172.21.15.88:0/694153911 >> 172.21.15.88:0/694153911 conn(0x7f9d380fa130 msgr2=0x7f9d380fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:46.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.541+0000 7f9d3fa9b700 1 -- 172.21.15.88:0/694153911 shutdown_connections 2023-07-14T17:42:46.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.542+0000 7f9d3fa9b700 1 -- 172.21.15.88:0/694153911 wait complete. 2023-07-14T17:42:46.532 INFO:teuthology.orchestra.run.smithi088.stderr:set pool 4 allow_ec_overwrites to true 2023-07-14T17:42:46.544 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-07-14T17:42:46.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.686+0000 7fddfc158700 1 -- 172.21.15.88:0/739070188 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddf40ffd40 msgr2=0x7fddf4100190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:46.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.686+0000 7fddfc158700 1 --2- 172.21.15.88:0/739070188 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddf40ffd40 0x7fddf4100190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fddf0009990 tx=0x7fddf0009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:46.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.686+0000 7fddfc158700 1 -- 172.21.15.88:0/739070188 shutdown_connections 2023-07-14T17:42:46.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.686+0000 7fddfc158700 1 --2- 172.21.15.88:0/739070188 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fddf41006d0 0x7fddf4106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:46.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.686+0000 7fddfc158700 1 --2- 172.21.15.88:0/739070188 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddf40feb50 0x7fddf40fef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:46.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.686+0000 7fddfc158700 1 --2- 172.21.15.88:0/739070188 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddf40ffd40 0x7fddf4100190 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:46.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.686+0000 7fddfc158700 1 -- 172.21.15.88:0/739070188 >> 172.21.15.88:0/739070188 conn(0x7fddf40fa150 msgr2=0x7fddf40fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:46.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.686+0000 7fddfc158700 1 -- 172.21.15.88:0/739070188 shutdown_connections 2023-07-14T17:42:46.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.686+0000 7fddfc158700 1 -- 172.21.15.88:0/739070188 wait complete. 2023-07-14T17:42:46.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.686+0000 7fddfc158700 1 Processor -- start 2023-07-14T17:42:46.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.686+0000 7fddfc158700 1 -- start start 2023-07-14T17:42:46.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.686+0000 7fddfc158700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddf40feb50 0x7fddf4102ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:46.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.686+0000 7fddfc158700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fddf40ffd40 0x7fddf4101120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:46.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.686+0000 7fddfc158700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddf41006d0 0x7fddf4101660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:46.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.686+0000 7fddfc158700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf4101e00 con 0x7fddf40feb50 2023-07-14T17:42:46.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.686+0000 7fddfc158700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf4101f70 con 0x7fddf41006d0 2023-07-14T17:42:46.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.686+0000 7fddfc158700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf419f950 con 0x7fddf40ffd40 2023-07-14T17:42:46.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.687+0000 7fddf9ef4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddf40feb50 0x7fddf4102ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:46.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.687+0000 7fddf9ef4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddf40feb50 0x7fddf4102ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41474/0 (socket says 172.21.15.88:41474) 2023-07-14T17:42:46.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.687+0000 7fddf9ef4700 1 -- 172.21.15.88:0/3417313528 learned_addr learned my addr 172.21.15.88:0/3417313528 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:46.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.687+0000 7fddfa6f5700 1 --2- 172.21.15.88:0/3417313528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddf41006d0 0x7fddf4101660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:46.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.687+0000 7fddf9ef4700 1 -- 172.21.15.88:0/3417313528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fddf40ffd40 msgr2=0x7fddf4101120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:46.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.687+0000 7fddf9ef4700 1 --2- 172.21.15.88:0/3417313528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fddf40ffd40 0x7fddf4101120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:46.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.687+0000 7fddf9ef4700 1 -- 172.21.15.88:0/3417313528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddf41006d0 msgr2=0x7fddf4101660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:46.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.687+0000 7fddf9ef4700 1 --2- 172.21.15.88:0/3417313528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddf41006d0 0x7fddf4101660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:46.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.687+0000 7fddf9ef4700 1 -- 172.21.15.88:0/3417313528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf00095f0 con 0x7fddf40feb50 2023-07-14T17:42:46.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.687+0000 7fddfa6f5700 1 --2- 172.21.15.88:0/3417313528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddf41006d0 0x7fddf4101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:42:46.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.687+0000 7fddf9ef4700 1 --2- 172.21.15.88:0/3417313528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddf40feb50 0x7fddf4102ae0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdde400c970 tx=0x7fdde400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:46.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.687+0000 7fddeaffd700 1 -- 172.21.15.88:0/3417313528 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde4004220 con 0x7fddf40feb50 2023-07-14T17:42:46.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.687+0000 7fddeaffd700 1 -- 172.21.15.88:0/3417313528 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdde4007d30 con 0x7fddf40feb50 2023-07-14T17:42:46.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.687+0000 7fddfc158700 1 -- 172.21.15.88:0/3417313528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf419fc30 con 0x7fddf40feb50 2023-07-14T17:42:46.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.687+0000 7fddeaffd700 1 -- 172.21.15.88:0/3417313528 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde4003bf0 con 0x7fddf40feb50 2023-07-14T17:42:46.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.687+0000 7fddfc158700 1 -- 172.21.15.88:0/3417313528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf41a0140 con 0x7fddf40feb50 2023-07-14T17:42:46.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.688+0000 7fddfc158700 1 -- 172.21.15.88:0/3417313528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf410a220 con 0x7fddf40feb50 2023-07-14T17:42:46.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.689+0000 7fddeaffd700 1 -- 172.21.15.88:0/3417313528 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fdde4003d70 con 0x7fddf40feb50 2023-07-14T17:42:46.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.690+0000 7fddeaffd700 1 --2- 172.21.15.88:0/3417313528 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fdde00a1150 0x7fdde00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:46.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.690+0000 7fddf96f3700 1 --2- 172.21.15.88:0/3417313528 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fdde00a1150 0x7fdde00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:46.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.690+0000 7fddeaffd700 1 -- 172.21.15.88:0/3417313528 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8698+0+0 (secure 0 0 0) 0x7fdde40ccb80 con 0x7fddf40feb50 2023-07-14T17:42:46.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.690+0000 7fddf96f3700 1 --2- 172.21.15.88:0/3417313528 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fdde00a1150 0x7fdde00a3600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fddf00095c0 tx=0x7fddf0009530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:46.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.694+0000 7fddeaffd700 1 -- 172.21.15.88:0/3417313528 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdde4096100 con 0x7fddf40feb50 2023-07-14T17:42:46.914 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:46 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1533539528' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T17:42:46.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:46 smithi178 ceph-mon[129368]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T17:42:46.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:46 smithi178 ceph-mon[129368]: pgmap v255: 129 pgs: 10 creating+peering, 33 unknown, 86 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2023-07-14T17:42:46.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:46 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/694153911' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-14T17:42:46.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:46.932+0000 7fddfc158700 1 -- 172.21.15.88:0/3417313528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fddf41a06f0 con 0x7fddf40feb50 2023-07-14T17:42:46.941 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:46 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1533539528' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T17:42:46.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:46 smithi088 ceph-mon[127465]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T17:42:46.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:46 smithi088 ceph-mon[127465]: pgmap v255: 129 pgs: 10 creating+peering, 33 unknown, 86 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2023-07-14T17:42:46.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:46 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/694153911' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-14T17:42:46.974 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:46 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1533539528' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T17:42:46.974 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:46 smithi107 ceph-mon[128999]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T17:42:46.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:46 smithi107 ceph-mon[128999]: pgmap v255: 129 pgs: 10 creating+peering, 33 unknown, 86 active+clean; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2023-07-14T17:42:46.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:46 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/694153911' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-14T17:42:47.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.733+0000 7fddeaffd700 1 -- 172.21.15.88:0/3417313528 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7fdde401d0b0 con 0x7fddf40feb50 2023-07-14T17:42:47.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.736+0000 7fddfc158700 1 -- 172.21.15.88:0/3417313528 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fdde00a1150 msgr2=0x7fdde00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:47.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.736+0000 7fddfc158700 1 --2- 172.21.15.88:0/3417313528 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fdde00a1150 0x7fdde00a3600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fddf00095c0 tx=0x7fddf0009530 comp rx=0 tx=0).stop 2023-07-14T17:42:47.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.736+0000 7fddfc158700 1 -- 172.21.15.88:0/3417313528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddf40feb50 msgr2=0x7fddf4102ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:47.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.736+0000 7fddfc158700 1 --2- 172.21.15.88:0/3417313528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddf40feb50 0x7fddf4102ae0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fdde400c970 tx=0x7fdde400cd30 comp rx=0 tx=0).stop 2023-07-14T17:42:47.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.736+0000 7fddfc158700 1 -- 172.21.15.88:0/3417313528 shutdown_connections 2023-07-14T17:42:47.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.736+0000 7fddfc158700 1 --2- 172.21.15.88:0/3417313528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fddf41006d0 0x7fddf4101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:47.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.736+0000 7fddfc158700 1 --2- 172.21.15.88:0/3417313528 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fdde00a1150 0x7fdde00a3600 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:47.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.736+0000 7fddfc158700 1 --2- 172.21.15.88:0/3417313528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fddf40feb50 0x7fddf4102ae0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:47.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.736+0000 7fddfc158700 1 --2- 172.21.15.88:0/3417313528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fddf40ffd40 0x7fddf4101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:47.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.736+0000 7fddfc158700 1 -- 172.21.15.88:0/3417313528 >> 172.21.15.88:0/3417313528 conn(0x7fddf40fa150 msgr2=0x7fddf40689f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:47.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.736+0000 7fddfc158700 1 -- 172.21.15.88:0/3417313528 shutdown_connections 2023-07-14T17:42:47.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.736+0000 7fddfc158700 1 -- 172.21.15.88:0/3417313528 wait complete. 2023-07-14T17:42:47.721 INFO:teuthology.orchestra.run.smithi088.stderr:added data pool 4 to fsmap 2023-07-14T17:42:47.734 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:42:47.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.880+0000 7fa61c2a0700 1 -- 172.21.15.88:0/1753896503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa614100180 msgr2=0x7fa614100590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:47.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.880+0000 7fa61c2a0700 1 --2- 172.21.15.88:0/1753896503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa614100180 0x7fa614100590 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa604009990 tx=0x7fa604009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:47.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.880+0000 7fa61c2a0700 1 -- 172.21.15.88:0/1753896503 shutdown_connections 2023-07-14T17:42:47.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.880+0000 7fa61c2a0700 1 --2- 172.21.15.88:0/1753896503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6140ff6f0 0x7fa614106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:47.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.880+0000 7fa61c2a0700 1 --2- 172.21.15.88:0/1753896503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa614100180 0x7fa614100590 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:47.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.880+0000 7fa61c2a0700 1 --2- 172.21.15.88:0/1753896503 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa6140fed40 0x7fa6140ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:47.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.880+0000 7fa61c2a0700 1 -- 172.21.15.88:0/1753896503 >> 172.21.15.88:0/1753896503 conn(0x7fa6140f98a0 msgr2=0x7fa6140fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:47.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.880+0000 7fa61c2a0700 1 -- 172.21.15.88:0/1753896503 shutdown_connections 2023-07-14T17:42:47.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.880+0000 7fa61c2a0700 1 -- 172.21.15.88:0/1753896503 wait complete. 2023-07-14T17:42:47.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.880+0000 7fa61c2a0700 1 Processor -- start 2023-07-14T17:42:47.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61c2a0700 1 -- start start 2023-07-14T17:42:47.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61c2a0700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa6140fed40 0x7fa614197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:47.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61c2a0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6140ff6f0 0x7fa614197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:47.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61c2a0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa614100180 0x7fa61419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:47.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61c2a0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61419f610 con 0x7fa614100180 2023-07-14T17:42:47.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61c2a0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61419f750 con 0x7fa6140ff6f0 2023-07-14T17:42:47.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61c2a0700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61419f8e0 con 0x7fa6140fed40 2023-07-14T17:42:47.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61a83d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa614100180 0x7fa61419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:47.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61983b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6140ff6f0 0x7fa614197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:47.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61a83d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa614100180 0x7fa61419f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41492/0 (socket says 172.21.15.88:41492) 2023-07-14T17:42:47.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61a83d700 1 -- 172.21.15.88:0/1481771963 learned_addr learned my addr 172.21.15.88:0/1481771963 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:47.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61a03c700 1 --2- 172.21.15.88:0/1481771963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa6140fed40 0x7fa614197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:47.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61a83d700 1 -- 172.21.15.88:0/1481771963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa6140fed40 msgr2=0x7fa614197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:47.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61a83d700 1 --2- 172.21.15.88:0/1481771963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa6140fed40 0x7fa614197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:47.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61a83d700 1 -- 172.21.15.88:0/1481771963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6140ff6f0 msgr2=0x7fa614197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:47.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61a83d700 1 --2- 172.21.15.88:0/1481771963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6140ff6f0 0x7fa614197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:47.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61a83d700 1 -- 172.21.15.88:0/1481771963 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6040095f0 con 0x7fa614100180 2023-07-14T17:42:47.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61a83d700 1 --2- 172.21.15.88:0/1481771963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa614100180 0x7fa61419f040 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa60c007ee0 tx=0x7fa60c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:47.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa60b7fe700 1 -- 172.21.15.88:0/1481771963 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c017070 con 0x7fa614100180 2023-07-14T17:42:47.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa60b7fe700 1 -- 172.21.15.88:0/1481771963 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa60c00f040 con 0x7fa614100180 2023-07-14T17:42:47.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.881+0000 7fa61c2a0700 1 -- 172.21.15.88:0/1481771963 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa61419fbc0 con 0x7fa614100180 2023-07-14T17:42:47.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.882+0000 7fa60b7fe700 1 -- 172.21.15.88:0/1481771963 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c005570 con 0x7fa614100180 2023-07-14T17:42:47.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.882+0000 7fa61c2a0700 1 -- 172.21.15.88:0/1481771963 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6141a0120 con 0x7fa614100180 2023-07-14T17:42:47.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.882+0000 7fa61c2a0700 1 -- 172.21.15.88:0/1481771963 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa614105ae0 con 0x7fa614100180 2023-07-14T17:42:47.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.884+0000 7fa60b7fe700 1 -- 172.21.15.88:0/1481771963 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa60c00db20 con 0x7fa614100180 2023-07-14T17:42:47.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.884+0000 7fa60b7fe700 1 --2- 172.21.15.88:0/1481771963 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa6000a1200 0x7fa6000a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:47.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.884+0000 7fa61a03c700 1 --2- 172.21.15.88:0/1481771963 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa6000a1200 0x7fa6000a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:47.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.884+0000 7fa60b7fe700 1 -- 172.21.15.88:0/1481771963 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa60c0cc7b0 con 0x7fa614100180 2023-07-14T17:42:47.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.885+0000 7fa61a03c700 1 --2- 172.21.15.88:0/1481771963 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa6000a1200 0x7fa6000a36b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa60400b5c0 tx=0x7fa604005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:47.871 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:47.887+0000 7fa60b7fe700 1 -- 172.21.15.88:0/1481771963 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa60c095d10 con 0x7fa614100180 2023-07-14T17:42:47.872 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:47 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/694153911' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-07-14T17:42:47.873 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:47 smithi107 ceph-mon[128999]: osdmap e85: 12 total, 12 up, 12 in 2023-07-14T17:42:47.873 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:47 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3417313528' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-07-14T17:42:47.914 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:47 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/694153911' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-07-14T17:42:47.914 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:47 smithi178 ceph-mon[129368]: osdmap e85: 12 total, 12 up, 12 in 2023-07-14T17:42:47.914 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:47 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3417313528' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-07-14T17:42:47.941 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:47 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/694153911' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-07-14T17:42:47.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:47 smithi088 ceph-mon[127465]: osdmap e85: 12 total, 12 up, 12 in 2023-07-14T17:42:47.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:47 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3417313528' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-07-14T17:42:48.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.117+0000 7fa61c2a0700 1 -- 172.21.15.88:0/1481771963 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa61402c8e0 con 0x7fa614100180 2023-07-14T17:42:48.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.118+0000 7fa60b7fe700 1 -- 172.21.15.88:0/1481771963 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10775 (secure 0 0 0) 0x7fa60c099330 con 0x7fa614100180 2023-07-14T17:42:48.101 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:48.101 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/3046592155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3046592155},{"type":"v1","addr":"172.21.15.88:6839","nonce":3046592155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6841/2509502918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6840","nonce":2509502918},{"type":"v1","addr":"172.21.15.88:6841","nonce":2509502918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/2332025068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":2332025068},{"type":"v1","addr":"172.21.15.178:6835","nonce":2332025068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3062637672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3062637672},{"type":"v1","addr":"172.21.15.107:6835","nonce":3062637672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/3950404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3950404084},{"type":"v1","addr":"172.21.15.178:6837","nonce":3950404084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24469,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/2953415077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2953415077},{"type":"v1","addr":"172.21.15.178:6839","nonce":2953415077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1152608615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1152608615},{"type":"v1","addr":"172.21.15.107:6837","nonce":1152608615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-14T17:42:42.314442+0000","modified":"2023-07-14T17:42:47.730622+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6839/31646548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":31646548},{"type":"v1","addr":"172.21.15.107:6839","nonce":31646548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-07-14T17:42:48.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.120+0000 7fa61c2a0700 1 -- 172.21.15.88:0/1481771963 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa6000a1200 msgr2=0x7fa6000a36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:48.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.120+0000 7fa61c2a0700 1 --2- 172.21.15.88:0/1481771963 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa6000a1200 0x7fa6000a36b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa60400b5c0 tx=0x7fa604005fb0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.103 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.120+0000 7fa61c2a0700 1 -- 172.21.15.88:0/1481771963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa614100180 msgr2=0x7fa61419f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:48.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.120+0000 7fa61c2a0700 1 --2- 172.21.15.88:0/1481771963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa614100180 0x7fa61419f040 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa60c007ee0 tx=0x7fa60c00d3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.120+0000 7fa61c2a0700 1 -- 172.21.15.88:0/1481771963 shutdown_connections 2023-07-14T17:42:48.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.120+0000 7fa61c2a0700 1 --2- 172.21.15.88:0/1481771963 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa6000a1200 0x7fa6000a36b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.120+0000 7fa61c2a0700 1 --2- 172.21.15.88:0/1481771963 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa614100180 0x7fa61419f040 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.120+0000 7fa61c2a0700 1 --2- 172.21.15.88:0/1481771963 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa6140fed40 0x7fa614197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.120+0000 7fa61c2a0700 1 --2- 172.21.15.88:0/1481771963 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6140ff6f0 0x7fa614197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.120+0000 7fa61c2a0700 1 -- 172.21.15.88:0/1481771963 >> 172.21.15.88:0/1481771963 conn(0x7fa6140f98a0 msgr2=0x7fa6140fbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:48.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.120+0000 7fa61c2a0700 1 -- 172.21.15.88:0/1481771963 shutdown_connections 2023-07-14T17:42:48.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.120+0000 7fa61c2a0700 1 -- 172.21.15.88:0/1481771963 wait complete. 2023-07-14T17:42:48.105 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 17 2023-07-14T17:42:48.115 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:42:48.224 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:42:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-b[154079]: 2023-07-14T17:42:47.888+0000 7fb592e7d700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-07-14T17:42:48.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.252+0000 7f8f67cfd700 1 -- 172.21.15.88:0/4164518423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f600fdef0 msgr2=0x7f8f600fe300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:48.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.252+0000 7f8f67cfd700 1 --2- 172.21.15.88:0/4164518423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f600fdef0 0x7f8f600fe300 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009990 tx=0x7f8f54009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.252+0000 7f8f67cfd700 1 -- 172.21.15.88:0/4164518423 shutdown_connections 2023-07-14T17:42:48.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.252+0000 7f8f67cfd700 1 --2- 172.21.15.88:0/4164518423 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f60101850 0x7f8f60108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.252+0000 7f8f67cfd700 1 --2- 172.21.15.88:0/4164518423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f600fdef0 0x7f8f600fe300 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.252+0000 7f8f67cfd700 1 --2- 172.21.15.88:0/4164518423 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f60100ec0 0x7f8f60101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.252+0000 7f8f67cfd700 1 -- 172.21.15.88:0/4164518423 >> 172.21.15.88:0/4164518423 conn(0x7f8f600f98b0 msgr2=0x7f8f600fbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:48.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.252+0000 7f8f67cfd700 1 -- 172.21.15.88:0/4164518423 shutdown_connections 2023-07-14T17:42:48.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.252+0000 7f8f67cfd700 1 -- 172.21.15.88:0/4164518423 wait complete. 2023-07-14T17:42:48.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.253+0000 7f8f67cfd700 1 Processor -- start 2023-07-14T17:42:48.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.253+0000 7f8f67cfd700 1 -- start start 2023-07-14T17:42:48.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.253+0000 7f8f67cfd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f600fdef0 0x7f8f60100830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:48.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.253+0000 7f8f67cfd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f60100ec0 0x7f8f600fee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:48.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.253+0000 7f8f67cfd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f60101850 0x7f8f600ff3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:48.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.253+0000 7f8f67cfd700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f600ffb50 con 0x7f8f600fdef0 2023-07-14T17:42:48.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.253+0000 7f8f67cfd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f600ffcc0 con 0x7f8f60100ec0 2023-07-14T17:42:48.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.253+0000 7f8f67cfd700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6019f880 con 0x7f8f60101850 2023-07-14T17:42:48.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.253+0000 7f8f65a99700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f600fdef0 0x7f8f60100830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:48.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.253+0000 7f8f65a99700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f600fdef0 0x7f8f60100830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41502/0 (socket says 172.21.15.88:41502) 2023-07-14T17:42:48.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.253+0000 7f8f65a99700 1 -- 172.21.15.88:0/3143887802 learned_addr learned my addr 172.21.15.88:0/3143887802 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:48.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.253+0000 7f8f65a99700 1 -- 172.21.15.88:0/3143887802 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f60101850 msgr2=0x7f8f600ff3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:48.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.253+0000 7f8f65a99700 1 --2- 172.21.15.88:0/3143887802 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f60101850 0x7f8f600ff3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.253+0000 7f8f65a99700 1 -- 172.21.15.88:0/3143887802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f60100ec0 msgr2=0x7f8f600fee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:48.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.253+0000 7f8f65298700 1 --2- 172.21.15.88:0/3143887802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f60100ec0 0x7f8f600fee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:48.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.253+0000 7f8f65a99700 1 --2- 172.21.15.88:0/3143887802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f60100ec0 0x7f8f600fee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.253+0000 7f8f65a99700 1 -- 172.21.15.88:0/3143887802 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f540095f0 con 0x7f8f600fdef0 2023-07-14T17:42:48.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.254+0000 7f8f65a99700 1 --2- 172.21.15.88:0/3143887802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f600fdef0 0x7f8f60100830 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8f54004730 tx=0x7f8f54003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:48.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.254+0000 7f8f52ffd700 1 -- 172.21.15.88:0/3143887802 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5401d070 con 0x7f8f600fdef0 2023-07-14T17:42:48.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.254+0000 7f8f52ffd700 1 -- 172.21.15.88:0/3143887802 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8f54004810 con 0x7f8f600fdef0 2023-07-14T17:42:48.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.254+0000 7f8f52ffd700 1 -- 172.21.15.88:0/3143887802 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54005320 con 0x7f8f600fdef0 2023-07-14T17:42:48.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.254+0000 7f8f67cfd700 1 -- 172.21.15.88:0/3143887802 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6019fb00 con 0x7f8f600fdef0 2023-07-14T17:42:48.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.254+0000 7f8f67cfd700 1 -- 172.21.15.88:0/3143887802 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6019ff20 con 0x7f8f600fdef0 2023-07-14T17:42:48.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.255+0000 7f8f67cfd700 1 -- 172.21.15.88:0/3143887802 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f60106c00 con 0x7f8f600fdef0 2023-07-14T17:42:48.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.256+0000 7f8f52ffd700 1 -- 172.21.15.88:0/3143887802 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8f54004980 con 0x7f8f600fdef0 2023-07-14T17:42:48.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.256+0000 7f8f52ffd700 1 --2- 172.21.15.88:0/3143887802 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8f4c0a1220 0x7f8f4c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:48.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.257+0000 7f8f65298700 1 --2- 172.21.15.88:0/3143887802 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8f4c0a1220 0x7f8f4c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:48.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.257+0000 7f8f52ffd700 1 -- 172.21.15.88:0/3143887802 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8f540ce070 con 0x7f8f600fdef0 2023-07-14T17:42:48.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.257+0000 7f8f65298700 1 --2- 172.21.15.88:0/3143887802 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8f4c0a1220 0x7f8f4c0a36d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c005950 tx=0x7f8f5c005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:48.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.260+0000 7f8f52ffd700 1 -- 172.21.15.88:0/3143887802 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8f54097660 con 0x7f8f600fdef0 2023-07-14T17:42:48.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.452+0000 7f8f67cfd700 1 -- 172.21.15.88:0/3143887802 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f6002cb50 con 0x7f8f600fdef0 2023-07-14T17:42:48.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.453+0000 7f8f52ffd700 1 -- 172.21.15.88:0/3143887802 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23985 (secure 0 0 0) 0x7f8f5409ac80 con 0x7f8f600fdef0 2023-07-14T17:42:48.437 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:48.437 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":86,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:42:46.934364+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:42:48.438 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.455+0000 7f8f67cfd700 1 -- 172.21.15.88:0/3143887802 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8f4c0a1220 msgr2=0x7f8f4c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:48.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.455+0000 7f8f67cfd700 1 --2- 172.21.15.88:0/3143887802 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8f4c0a1220 0x7f8f4c0a36d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c005950 tx=0x7f8f5c005cf0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.455+0000 7f8f67cfd700 1 -- 172.21.15.88:0/3143887802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f600fdef0 msgr2=0x7f8f60100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:48.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.455+0000 7f8f67cfd700 1 --2- 172.21.15.88:0/3143887802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f600fdef0 0x7f8f60100830 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8f54004730 tx=0x7f8f54003820 comp rx=0 tx=0).stop 2023-07-14T17:42:48.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.455+0000 7f8f67cfd700 1 -- 172.21.15.88:0/3143887802 shutdown_connections 2023-07-14T17:42:48.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.455+0000 7f8f67cfd700 1 --2- 172.21.15.88:0/3143887802 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8f60101850 0x7f8f600ff3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.455+0000 7f8f67cfd700 1 --2- 172.21.15.88:0/3143887802 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8f4c0a1220 0x7f8f4c0a36d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.455+0000 7f8f67cfd700 1 --2- 172.21.15.88:0/3143887802 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8f600fdef0 0x7f8f60100830 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.455+0000 7f8f67cfd700 1 --2- 172.21.15.88:0/3143887802 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8f60100ec0 0x7f8f600fee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.455+0000 7f8f67cfd700 1 -- 172.21.15.88:0/3143887802 >> 172.21.15.88:0/3143887802 conn(0x7f8f600f98b0 msgr2=0x7f8f600fbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:48.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.455+0000 7f8f67cfd700 1 -- 172.21.15.88:0/3143887802 shutdown_connections 2023-07-14T17:42:48.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.455+0000 7f8f67cfd700 1 -- 172.21.15.88:0/3143887802 wait complete. 2023-07-14T17:42:48.451 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:42:48.575 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.590+0000 7f8b3a849700 1 -- 172.21.15.88:0/2522504220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b340ffcf0 msgr2=0x7f8b34100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.590+0000 7f8b3a849700 1 --2- 172.21.15.88:0/2522504220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b340ffcf0 0x7f8b34100140 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8b24009990 tx=0x7f8b24009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.590+0000 7f8b3a849700 1 -- 172.21.15.88:0/2522504220 shutdown_connections 2023-07-14T17:42:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.590+0000 7f8b3a849700 1 --2- 172.21.15.88:0/2522504220 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b34100680 0x7f8b34106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.576 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.590+0000 7f8b3a849700 1 --2- 172.21.15.88:0/2522504220 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b340feb00 0x7f8b340fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.590+0000 7f8b3a849700 1 --2- 172.21.15.88:0/2522504220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b340ffcf0 0x7f8b34100140 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.590+0000 7f8b3a849700 1 -- 172.21.15.88:0/2522504220 >> 172.21.15.88:0/2522504220 conn(0x7f8b340fa100 msgr2=0x7f8b340fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.590+0000 7f8b3a849700 1 -- 172.21.15.88:0/2522504220 shutdown_connections 2023-07-14T17:42:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.590+0000 7f8b3a849700 1 -- 172.21.15.88:0/2522504220 wait complete. 2023-07-14T17:42:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.590+0000 7f8b3a849700 1 Processor -- start 2023-07-14T17:42:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.590+0000 7f8b3a849700 1 -- start start 2023-07-14T17:42:48.577 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b3a849700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b340feb00 0x7f8b34193650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b3a849700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b340ffcf0 0x7f8b34193b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b3a849700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b34100680 0x7f8b3419ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b3a849700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3419b1d0 con 0x7f8b340feb00 2023-07-14T17:42:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b3a849700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3419b310 con 0x7f8b34100680 2023-07-14T17:42:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b3a849700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3419b4a0 con 0x7f8b340ffcf0 2023-07-14T17:42:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b33fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b340feb00 0x7f8b34193650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:48.578 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b33fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b340feb00 0x7f8b34193650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41516/0 (socket says 172.21.15.88:41516) 2023-07-14T17:42:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b33fff700 1 -- 172.21.15.88:0/3561451689 learned_addr learned my addr 172.21.15.88:0/3561451689 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b337fe700 1 --2- 172.21.15.88:0/3561451689 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b340ffcf0 0x7f8b34193b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b38de6700 1 --2- 172.21.15.88:0/3561451689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b34100680 0x7f8b3419ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b33fff700 1 -- 172.21.15.88:0/3561451689 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b340ffcf0 msgr2=0x7f8b34193b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b33fff700 1 --2- 172.21.15.88:0/3561451689 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b340ffcf0 0x7f8b34193b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b33fff700 1 -- 172.21.15.88:0/3561451689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b34100680 msgr2=0x7f8b3419ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:48.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b33fff700 1 --2- 172.21.15.88:0/3561451689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b34100680 0x7f8b3419ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b33fff700 1 -- 172.21.15.88:0/3561451689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b240095f0 con 0x7f8b340feb00 2023-07-14T17:42:48.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b38de6700 1 --2- 172.21.15.88:0/3561451689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b34100680 0x7f8b3419ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:42:48.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b33fff700 1 --2- 172.21.15.88:0/3561451689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b340feb00 0x7f8b34193650 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800d980 tx=0x7f8b2800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:48.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b317fa700 1 -- 172.21.15.88:0/3561451689 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b28004020 con 0x7f8b340feb00 2023-07-14T17:42:48.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b317fa700 1 -- 172.21.15.88:0/3561451689 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8b28009730 con 0x7f8b340feb00 2023-07-14T17:42:48.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b3a849700 1 -- 172.21.15.88:0/3561451689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b3419b780 con 0x7f8b340feb00 2023-07-14T17:42:48.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b317fa700 1 -- 172.21.15.88:0/3561451689 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b28003a00 con 0x7f8b340feb00 2023-07-14T17:42:48.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.591+0000 7f8b3a849700 1 -- 172.21.15.88:0/3561451689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3419bce0 con 0x7f8b340feb00 2023-07-14T17:42:48.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.593+0000 7f8b3a849700 1 -- 172.21.15.88:0/3561451689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b34106090 con 0x7f8b340feb00 2023-07-14T17:42:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.594+0000 7f8b317fa700 1 -- 172.21.15.88:0/3561451689 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8b280098a0 con 0x7f8b340feb00 2023-07-14T17:42:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.594+0000 7f8b317fa700 1 --2- 172.21.15.88:0/3561451689 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8b1c0a1220 0x7f8b1c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.594+0000 7f8b337fe700 1 --2- 172.21.15.88:0/3561451689 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8b1c0a1220 0x7f8b1c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.594+0000 7f8b317fa700 1 -- 172.21.15.88:0/3561451689 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8b280ccae0 con 0x7f8b340feb00 2023-07-14T17:42:48.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.595+0000 7f8b337fe700 1 --2- 172.21.15.88:0/3561451689 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8b1c0a1220 0x7f8b1c0a36d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8b24000c00 tx=0x7f8b24005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:48.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.597+0000 7f8b317fa700 1 -- 172.21.15.88:0/3561451689 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8b280960c0 con 0x7f8b340feb00 2023-07-14T17:42:48.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.793+0000 7f8b3a849700 1 -- 172.21.15.88:0/3561451689 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b3402cb50 con 0x7f8b340feb00 2023-07-14T17:42:48.777 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.794+0000 7f8b317fa700 1 -- 172.21.15.88:0/3561451689 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23985 (secure 0 0 0) 0x7f8b280996e0 con 0x7f8b340feb00 2023-07-14T17:42:48.778 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:48.778 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":86,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:42:46.934364+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:42:48.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.796+0000 7f8b3a849700 1 -- 172.21.15.88:0/3561451689 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8b1c0a1220 msgr2=0x7f8b1c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:48.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.796+0000 7f8b3a849700 1 --2- 172.21.15.88:0/3561451689 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8b1c0a1220 0x7f8b1c0a36d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8b24000c00 tx=0x7f8b24005fd0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.796+0000 7f8b3a849700 1 -- 172.21.15.88:0/3561451689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b340feb00 msgr2=0x7f8b34193650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.796+0000 7f8b3a849700 1 --2- 172.21.15.88:0/3561451689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b340feb00 0x7f8b34193650 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800d980 tx=0x7f8b2800dc90 comp rx=0 tx=0).stop 2023-07-14T17:42:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.796+0000 7f8b3a849700 1 -- 172.21.15.88:0/3561451689 shutdown_connections 2023-07-14T17:42:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.796+0000 7f8b3a849700 1 --2- 172.21.15.88:0/3561451689 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b34100680 0x7f8b3419ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.796+0000 7f8b3a849700 1 --2- 172.21.15.88:0/3561451689 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8b1c0a1220 0x7f8b1c0a36d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.796+0000 7f8b3a849700 1 --2- 172.21.15.88:0/3561451689 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b340feb00 0x7f8b34193650 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.796+0000 7f8b3a849700 1 --2- 172.21.15.88:0/3561451689 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b340ffcf0 0x7f8b34193b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.796+0000 7f8b3a849700 1 -- 172.21.15.88:0/3561451689 >> 172.21.15.88:0/3561451689 conn(0x7f8b340fa100 msgr2=0x7f8b341031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:48.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.796+0000 7f8b3a849700 1 -- 172.21.15.88:0/3561451689 shutdown_connections 2023-07-14T17:42:48.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.796+0000 7f8b3a849700 1 -- 172.21.15.88:0/3561451689 wait complete. 2023-07-14T17:42:48.790 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:48 smithi088 ceph-mon[127465]: osdmap e86: 12 total, 12 up, 12 in 2023-07-14T17:42:48.790 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:48 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3417313528' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-07-14T17:42:48.791 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:48 smithi088 ceph-mon[127465]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T17:42:48.791 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:48 smithi088 ceph-mon[127465]: pgmap v258: 129 pgs: 8 creating+peering, 19 unknown, 102 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 4 op/s 2023-07-14T17:42:48.791 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:48 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1481771963' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:48.791 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:48 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3143887802' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:48.809 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:42:48.914 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:48 smithi178 ceph-mon[129368]: osdmap e86: 12 total, 12 up, 12 in 2023-07-14T17:42:48.914 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:48 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3417313528' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-07-14T17:42:48.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:48 smithi178 ceph-mon[129368]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T17:42:48.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:48 smithi178 ceph-mon[129368]: pgmap v258: 129 pgs: 8 creating+peering, 19 unknown, 102 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 4 op/s 2023-07-14T17:42:48.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:48 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1481771963' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:48.915 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:48 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3143887802' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:48.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.947+0000 7f8b1f8a0700 1 -- 172.21.15.88:0/1528174328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b180ff6b0 msgr2=0x7f8b18106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:48.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.947+0000 7f8b1f8a0700 1 --2- 172.21.15.88:0/1528174328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b180ff6b0 0x7f8b18106f60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8b1000b0a0 tx=0x7f8b1000b3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.947+0000 7f8b1f8a0700 1 -- 172.21.15.88:0/1528174328 shutdown_connections 2023-07-14T17:42:48.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.947+0000 7f8b1f8a0700 1 --2- 172.21.15.88:0/1528174328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b180ff6b0 0x7f8b18106f60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.947+0000 7f8b1f8a0700 1 --2- 172.21.15.88:0/1528174328 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b181000b0 0x7f8b181004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.947+0000 7f8b1f8a0700 1 --2- 172.21.15.88:0/1528174328 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b180fed00 0x7f8b180ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.947+0000 7f8b1f8a0700 1 -- 172.21.15.88:0/1528174328 >> 172.21.15.88:0/1528174328 conn(0x7f8b180f98a0 msgr2=0x7f8b180fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:48.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.948+0000 7f8b1f8a0700 1 -- 172.21.15.88:0/1528174328 shutdown_connections 2023-07-14T17:42:48.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.948+0000 7f8b1f8a0700 1 -- 172.21.15.88:0/1528174328 wait complete. 2023-07-14T17:42:48.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.948+0000 7f8b1f8a0700 1 Processor -- start 2023-07-14T17:42:48.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.948+0000 7f8b1f8a0700 1 -- start start 2023-07-14T17:42:48.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.948+0000 7f8b1f8a0700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b180fed00 0x7f8b180fe7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:48.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.948+0000 7f8b1f8a0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b180ff6b0 0x7f8b1819a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:48.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.948+0000 7f8b1f8a0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b181000b0 0x7f8b1819cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:48.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.948+0000 7f8b1f8a0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1819d410 con 0x7f8b180ff6b0 2023-07-14T17:42:48.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.948+0000 7f8b1f8a0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1819d580 con 0x7f8b181000b0 2023-07-14T17:42:48.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.948+0000 7f8b1f8a0700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1819d740 con 0x7f8b180fed00 2023-07-14T17:42:48.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.948+0000 7f8b1ce3b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b180ff6b0 0x7f8b1819a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:48.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.948+0000 7f8b1ce3b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b180ff6b0 0x7f8b1819a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41532/0 (socket says 172.21.15.88:41532) 2023-07-14T17:42:48.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.948+0000 7f8b1ce3b700 1 -- 172.21.15.88:0/2896787877 learned_addr learned my addr 172.21.15.88:0/2896787877 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:48.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.948+0000 7f8b1d63c700 1 --2- 172.21.15.88:0/2896787877 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b180fed00 0x7f8b180fe7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:48.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.949+0000 7f8b1de3d700 1 --2- 172.21.15.88:0/2896787877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b181000b0 0x7f8b1819cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:48.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.949+0000 7f8b1ce3b700 1 -- 172.21.15.88:0/2896787877 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b180fed00 msgr2=0x7f8b180fe7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:48.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.949+0000 7f8b1ce3b700 1 --2- 172.21.15.88:0/2896787877 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b180fed00 0x7f8b180fe7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.949+0000 7f8b1ce3b700 1 -- 172.21.15.88:0/2896787877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b181000b0 msgr2=0x7f8b1819cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:48.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.949+0000 7f8b1ce3b700 1 --2- 172.21.15.88:0/2896787877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b181000b0 0x7f8b1819cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:48.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.949+0000 7f8b1ce3b700 1 -- 172.21.15.88:0/2896787877 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b10009d00 con 0x7f8b180ff6b0 2023-07-14T17:42:48.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.949+0000 7f8b1d63c700 1 --2- 172.21.15.88:0/2896787877 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b180fed00 0x7f8b180fe7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:42:48.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.949+0000 7f8b1ce3b700 1 --2- 172.21.15.88:0/2896787877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b180ff6b0 0x7f8b1819a5e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8b0800e9a0 tx=0x7f8b0800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:48.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.949+0000 7f8b0e7fc700 1 -- 172.21.15.88:0/2896787877 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b08009790 con 0x7f8b180ff6b0 2023-07-14T17:42:48.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.949+0000 7f8b1f8a0700 1 -- 172.21.15.88:0/2896787877 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b1819da20 con 0x7f8b180ff6b0 2023-07-14T17:42:48.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.949+0000 7f8b0e7fc700 1 -- 172.21.15.88:0/2896787877 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8b0800cbd0 con 0x7f8b180ff6b0 2023-07-14T17:42:48.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.949+0000 7f8b0e7fc700 1 -- 172.21.15.88:0/2896787877 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b08018680 con 0x7f8b180ff6b0 2023-07-14T17:42:48.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.949+0000 7f8b1f8a0700 1 -- 172.21.15.88:0/2896787877 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b1819df80 con 0x7f8b180ff6b0 2023-07-14T17:42:48.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.950+0000 7f8b1f8a0700 1 -- 172.21.15.88:0/2896787877 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b18106130 con 0x7f8b180ff6b0 2023-07-14T17:42:48.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.951+0000 7f8b0e7fc700 1 -- 172.21.15.88:0/2896787877 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8b0800cd40 con 0x7f8b180ff6b0 2023-07-14T17:42:48.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.951+0000 7f8b0e7fc700 1 --2- 172.21.15.88:0/2896787877 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8b040a1220 0x7f8b040a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:48.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.952+0000 7f8b1d63c700 1 --2- 172.21.15.88:0/2896787877 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8b040a1220 0x7f8b040a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:48.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.952+0000 7f8b0e7fc700 1 -- 172.21.15.88:0/2896787877 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8b08014070 con 0x7f8b180ff6b0 2023-07-14T17:42:48.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.952+0000 7f8b1d63c700 1 --2- 172.21.15.88:0/2896787877 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8b040a1220 0x7f8b040a36d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8b14005950 tx=0x7f8b140058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:48.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:48.955+0000 7f8b0e7fc700 1 -- 172.21.15.88:0/2896787877 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8b080965b0 con 0x7f8b180ff6b0 2023-07-14T17:42:48.974 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:48 smithi107 ceph-mon[128999]: osdmap e86: 12 total, 12 up, 12 in 2023-07-14T17:42:48.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:48 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3417313528' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-07-14T17:42:48.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:48 smithi107 ceph-mon[128999]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T17:42:48.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:48 smithi107 ceph-mon[128999]: pgmap v258: 129 pgs: 8 creating+peering, 19 unknown, 102 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 4 op/s 2023-07-14T17:42:48.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:48 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1481771963' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:48.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:48 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3143887802' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:49.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.187+0000 7f8b1f8a0700 1 -- 172.21.15.88:0/2896787877 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b1802c8e0 con 0x7f8b180ff6b0 2023-07-14T17:42:49.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.188+0000 7f8b0e7fc700 1 -- 172.21.15.88:0/2896787877 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10775 (secure 0 0 0) 0x7f8b08026090 con 0x7f8b180ff6b0 2023-07-14T17:42:49.171 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:49.171 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/3046592155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3046592155},{"type":"v1","addr":"172.21.15.88:6839","nonce":3046592155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6841/2509502918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6840","nonce":2509502918},{"type":"v1","addr":"172.21.15.88:6841","nonce":2509502918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24415,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6835/2332025068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":2332025068},{"type":"v1","addr":"172.21.15.178:6835","nonce":2332025068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6835/3062637672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3062637672},{"type":"v1","addr":"172.21.15.107:6835","nonce":3062637672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6837/3950404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3950404084},{"type":"v1","addr":"172.21.15.178:6837","nonce":3950404084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24469,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.178:6839/2953415077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2953415077},{"type":"v1","addr":"172.21.15.178:6839","nonce":2953415077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24473,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.107:6837/1152608615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1152608615},{"type":"v1","addr":"172.21.15.107:6837","nonce":1152608615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-14T17:42:42.314442+0000","modified":"2023-07-14T17:42:47.730622+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24506},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6839/31646548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":31646548},{"type":"v1","addr":"172.21.15.107:6839","nonce":31646548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-07-14T17:42:49.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.190+0000 7f8b1f8a0700 1 -- 172.21.15.88:0/2896787877 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8b040a1220 msgr2=0x7f8b040a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:49.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.190+0000 7f8b1f8a0700 1 --2- 172.21.15.88:0/2896787877 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8b040a1220 0x7f8b040a36d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8b14005950 tx=0x7f8b140058e0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.190+0000 7f8b1f8a0700 1 -- 172.21.15.88:0/2896787877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b180ff6b0 msgr2=0x7f8b1819a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:49.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.190+0000 7f8b1f8a0700 1 --2- 172.21.15.88:0/2896787877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b180ff6b0 0x7f8b1819a5e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8b0800e9a0 tx=0x7f8b0800ed60 comp rx=0 tx=0).stop 2023-07-14T17:42:49.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.190+0000 7f8b1f8a0700 1 -- 172.21.15.88:0/2896787877 shutdown_connections 2023-07-14T17:42:49.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.190+0000 7f8b1f8a0700 1 --2- 172.21.15.88:0/2896787877 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8b181000b0 0x7f8b1819cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.190+0000 7f8b1f8a0700 1 --2- 172.21.15.88:0/2896787877 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8b180fed00 0x7f8b180fe7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.190+0000 7f8b1f8a0700 1 --2- 172.21.15.88:0/2896787877 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8b040a1220 0x7f8b040a36d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.190+0000 7f8b1f8a0700 1 --2- 172.21.15.88:0/2896787877 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b180ff6b0 0x7f8b1819a5e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.190+0000 7f8b1f8a0700 1 -- 172.21.15.88:0/2896787877 >> 172.21.15.88:0/2896787877 conn(0x7f8b180f98a0 msgr2=0x7f8b18103270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:49.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.190+0000 7f8b1f8a0700 1 -- 172.21.15.88:0/2896787877 shutdown_connections 2023-07-14T17:42:49.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.190+0000 7f8b1f8a0700 1 -- 172.21.15.88:0/2896787877 wait complete. 2023-07-14T17:42:49.175 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 17 2023-07-14T17:42:49.186 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:42:49.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.324+0000 7f578d8f6700 1 -- 172.21.15.88:0/3934064289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57880ff740 msgr2=0x7f5788106ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:49.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.324+0000 7f578d8f6700 1 --2- 172.21.15.88:0/3934064289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57880ff740 0x7f5788106ff0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f578000b0a0 tx=0x7f578000b3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.324+0000 7f578d8f6700 1 -- 172.21.15.88:0/3934064289 shutdown_connections 2023-07-14T17:42:49.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.324+0000 7f578d8f6700 1 --2- 172.21.15.88:0/3934064289 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57880ff740 0x7f5788106ff0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.324+0000 7f578d8f6700 1 --2- 172.21.15.88:0/3934064289 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f57881001d0 0x7f57881005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.324+0000 7f578d8f6700 1 --2- 172.21.15.88:0/3934064289 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880fed90 0x7f57880ff200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.324+0000 7f578d8f6700 1 -- 172.21.15.88:0/3934064289 >> 172.21.15.88:0/3934064289 conn(0x7f57880f98d0 msgr2=0x7f57880fbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:49.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.324+0000 7f578d8f6700 1 -- 172.21.15.88:0/3934064289 shutdown_connections 2023-07-14T17:42:49.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.324+0000 7f578d8f6700 1 -- 172.21.15.88:0/3934064289 wait complete. 2023-07-14T17:42:49.309 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.324+0000 7f578d8f6700 1 Processor -- start 2023-07-14T17:42:49.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f578d8f6700 1 -- start start 2023-07-14T17:42:49.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f578d8f6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880fed90 0x7f5788195870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f578d8f6700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57880ff740 0x7f5788195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f578d8f6700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f57881001d0 0x7f578819ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f578d8f6700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578819d3f0 con 0x7f57880ff740 2023-07-14T17:42:49.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f578d8f6700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578819d530 con 0x7f57880fed90 2023-07-14T17:42:49.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f578d8f6700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578819d6c0 con 0x7f57881001d0 2023-07-14T17:42:49.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f57867fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57880ff740 0x7f5788195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f57867fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57880ff740 0x7f5788195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41546/0 (socket says 172.21.15.88:41546) 2023-07-14T17:42:49.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f57867fc700 1 -- 172.21.15.88:0/1247638051 learned_addr learned my addr 172.21.15.88:0/1247638051 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:49.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f5786ffd700 1 --2- 172.21.15.88:0/1247638051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880fed90 0x7f5788195870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f57877fe700 1 --2- 172.21.15.88:0/1247638051 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f57881001d0 0x7f578819ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f57867fc700 1 -- 172.21.15.88:0/1247638051 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f57881001d0 msgr2=0x7f578819ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:49.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f57867fc700 1 --2- 172.21.15.88:0/1247638051 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f57881001d0 0x7f578819ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f57867fc700 1 -- 172.21.15.88:0/1247638051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880fed90 msgr2=0x7f5788195870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f57867fc700 1 --2- 172.21.15.88:0/1247638051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880fed90 0x7f5788195870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f57867fc700 1 -- 172.21.15.88:0/1247638051 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5780009d00 con 0x7f57880ff740 2023-07-14T17:42:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f57867fc700 1 --2- 172.21.15.88:0/1247638051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57880ff740 0x7f5788195db0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f577800d950 tx=0x7f577800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f578c8f4700 1 -- 172.21.15.88:0/1247638051 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577800f890 con 0x7f57880ff740 2023-07-14T17:42:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f578d8f6700 1 -- 172.21.15.88:0/1247638051 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f578819d9a0 con 0x7f57880ff740 2023-07-14T17:42:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f578c8f4700 1 -- 172.21.15.88:0/1247638051 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f577800fe80 con 0x7f57880ff740 2023-07-14T17:42:49.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.325+0000 7f578c8f4700 1 -- 172.21.15.88:0/1247638051 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577800e3b0 con 0x7f57880ff740 2023-07-14T17:42:49.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.326+0000 7f578d8f6700 1 -- 172.21.15.88:0/1247638051 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f578819df00 con 0x7f57880ff740 2023-07-14T17:42:49.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.326+0000 7f578d8f6700 1 -- 172.21.15.88:0/1247638051 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f578810a1d0 con 0x7f57880ff740 2023-07-14T17:42:49.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.328+0000 7f578c8f4700 1 -- 172.21.15.88:0/1247638051 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5778010460 con 0x7f57880ff740 2023-07-14T17:42:49.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.328+0000 7f578c8f4700 1 --2- 172.21.15.88:0/1247638051 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f57700a11a0 0x7f57700a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.328+0000 7f5786ffd700 1 --2- 172.21.15.88:0/1247638051 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f57700a11a0 0x7f57700a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.328+0000 7f578c8f4700 1 -- 172.21.15.88:0/1247638051 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f57780ccbf0 con 0x7f57880ff740 2023-07-14T17:42:49.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.328+0000 7f5786ffd700 1 --2- 172.21.15.88:0/1247638051 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f57700a11a0 0x7f57700a3650 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f577c005950 tx=0x7f577c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:49.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.331+0000 7f578c8f4700 1 -- 172.21.15.88:0/1247638051 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5778096150 con 0x7f57880ff740 2023-07-14T17:42:49.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.523+0000 7f578d8f6700 1 -- 172.21.15.88:0/1247638051 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f578810a390 con 0x7f57880ff740 2023-07-14T17:42:49.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.525+0000 7f578c8f4700 1 -- 172.21.15.88:0/1247638051 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23985 (secure 0 0 0) 0x7f5778020090 con 0x7f57880ff740 2023-07-14T17:42:49.508 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:49.509 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":86,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:42:46.934364+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:42:49.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.527+0000 7f578d8f6700 1 -- 172.21.15.88:0/1247638051 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f57700a11a0 msgr2=0x7f57700a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:49.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.527+0000 7f578d8f6700 1 --2- 172.21.15.88:0/1247638051 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f57700a11a0 0x7f57700a3650 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f577c005950 tx=0x7f577c00a300 comp rx=0 tx=0).stop 2023-07-14T17:42:49.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.527+0000 7f578d8f6700 1 -- 172.21.15.88:0/1247638051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57880ff740 msgr2=0x7f5788195db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:49.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.527+0000 7f578d8f6700 1 --2- 172.21.15.88:0/1247638051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57880ff740 0x7f5788195db0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f577800d950 tx=0x7f577800dc60 comp rx=0 tx=0).stop 2023-07-14T17:42:49.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.527+0000 7f578d8f6700 1 -- 172.21.15.88:0/1247638051 shutdown_connections 2023-07-14T17:42:49.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.527+0000 7f578d8f6700 1 --2- 172.21.15.88:0/1247638051 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f57881001d0 0x7f578819ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.527+0000 7f578d8f6700 1 --2- 172.21.15.88:0/1247638051 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f57880fed90 0x7f5788195870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.527+0000 7f578d8f6700 1 --2- 172.21.15.88:0/1247638051 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f57700a11a0 0x7f57700a3650 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.527+0000 7f578d8f6700 1 --2- 172.21.15.88:0/1247638051 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f57880ff740 0x7f5788195db0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.527+0000 7f578d8f6700 1 -- 172.21.15.88:0/1247638051 >> 172.21.15.88:0/1247638051 conn(0x7f57880f98d0 msgr2=0x7f57880fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:49.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.527+0000 7f578d8f6700 1 -- 172.21.15.88:0/1247638051 shutdown_connections 2023-07-14T17:42:49.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.527+0000 7f578d8f6700 1 -- 172.21.15.88:0/1247638051 wait complete. 2023-07-14T17:42:49.523 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-07-14T17:42:42.314442+0000', 'modified': '2023-07-14T17:42:47.730622+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24506}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24506': {'gid': 24506, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.107:6839/31646548', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6838', 'nonce': 31646548}, {'type': 'v1', 'addr': '172.21.15.107:6839', 'nonce': 31646548}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-07-14T17:42:49.523 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-07-14T17:42:49.523 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-14T17:42:49.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.661+0000 7f1689952700 1 -- 172.21.15.88:0/289701357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1684100770 msgr2=0x7f1684102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:49.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.661+0000 7f1689952700 1 --2- 172.21.15.88:0/289701357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1684100770 0x7f1684102b50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f1674009990 tx=0x7f1674009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.661+0000 7f1689952700 1 -- 172.21.15.88:0/289701357 shutdown_connections 2023-07-14T17:42:49.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.661+0000 7f1689952700 1 --2- 172.21.15.88:0/289701357 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16841059b0 0x7f1684107d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.661+0000 7f1689952700 1 --2- 172.21.15.88:0/289701357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1684100770 0x7f1684102b50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.661+0000 7f1689952700 1 --2- 172.21.15.88:0/289701357 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1684103090 0x7f1684105470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.661+0000 7f1689952700 1 -- 172.21.15.88:0/289701357 >> 172.21.15.88:0/289701357 conn(0x7f16840fa150 msgr2=0x7f16840fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:49.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.661+0000 7f1689952700 1 -- 172.21.15.88:0/289701357 shutdown_connections 2023-07-14T17:42:49.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.661+0000 7f1689952700 1 -- 172.21.15.88:0/289701357 wait complete. 2023-07-14T17:42:49.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.662+0000 7f1689952700 1 Processor -- start 2023-07-14T17:42:49.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.662+0000 7f1689952700 1 -- start start 2023-07-14T17:42:49.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.662+0000 7f1689952700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1684100770 0x7f1684195870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.662+0000 7f1689952700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1684103090 0x7f1684195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.662+0000 7f1689952700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16841059b0 0x7f168419ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.662+0000 7f1689952700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168419d3f0 con 0x7f1684100770 2023-07-14T17:42:49.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.662+0000 7f1689952700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168419d530 con 0x7f16841059b0 2023-07-14T17:42:49.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.662+0000 7f1689952700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168419d6c0 con 0x7f1684103090 2023-07-14T17:42:49.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.662+0000 7f1682ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1684100770 0x7f1684195870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.662+0000 7f1682ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1684100770 0x7f1684195870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41558/0 (socket says 172.21.15.88:41558) 2023-07-14T17:42:49.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.662+0000 7f1682ffd700 1 -- 172.21.15.88:0/2282327594 learned_addr learned my addr 172.21.15.88:0/2282327594 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:49.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.662+0000 7f16827fc700 1 --2- 172.21.15.88:0/2282327594 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1684103090 0x7f1684195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.662+0000 7f1682ffd700 1 -- 172.21.15.88:0/2282327594 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1684103090 msgr2=0x7f1684195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:49.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.662+0000 7f1682ffd700 1 --2- 172.21.15.88:0/2282327594 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1684103090 0x7f1684195db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.662+0000 7f1682ffd700 1 -- 172.21.15.88:0/2282327594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16841059b0 msgr2=0x7f168419ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.662+0000 7f16837fe700 1 --2- 172.21.15.88:0/2282327594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16841059b0 0x7f168419ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.663+0000 7f1682ffd700 1 --2- 172.21.15.88:0/2282327594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16841059b0 0x7f168419ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.663+0000 7f1682ffd700 1 -- 172.21.15.88:0/2282327594 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16740095f0 con 0x7f1684100770 2023-07-14T17:42:49.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.663+0000 7f16837fe700 1 --2- 172.21.15.88:0/2282327594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16841059b0 0x7f168419ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:42:49.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.663+0000 7f1682ffd700 1 --2- 172.21.15.88:0/2282327594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1684100770 0x7f1684195870 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f16740098e0 tx=0x7f16740039d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:49.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.663+0000 7f1688950700 1 -- 172.21.15.88:0/2282327594 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167401d070 con 0x7f1684100770 2023-07-14T17:42:49.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.663+0000 7f1688950700 1 -- 172.21.15.88:0/2282327594 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1674004830 con 0x7f1684100770 2023-07-14T17:42:49.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.663+0000 7f1689952700 1 -- 172.21.15.88:0/2282327594 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168419d940 con 0x7f1684100770 2023-07-14T17:42:49.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.663+0000 7f1688950700 1 -- 172.21.15.88:0/2282327594 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16740053a0 con 0x7f1684100770 2023-07-14T17:42:49.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.663+0000 7f1689952700 1 -- 172.21.15.88:0/2282327594 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f168419dd00 con 0x7f1684100770 2023-07-14T17:42:49.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.665+0000 7f1688950700 1 -- 172.21.15.88:0/2282327594 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f16740049a0 con 0x7f1684100770 2023-07-14T17:42:49.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.666+0000 7f1688950700 1 --2- 172.21.15.88:0/2282327594 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f166c0a1150 0x7f166c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.666+0000 7f16827fc700 1 --2- 172.21.15.88:0/2282327594 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f166c0a1150 0x7f166c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.666+0000 7f1688950700 1 -- 172.21.15.88:0/2282327594 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f16740cdf10 con 0x7f1684100770 2023-07-14T17:42:49.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.666+0000 7f16827fc700 1 --2- 172.21.15.88:0/2282327594 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f166c0a1150 0x7f166c0a3600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1678005950 tx=0x7f16780058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:49.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.667+0000 7f1695aeab80 1 Processor -- start 2023-07-14T17:42:49.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.667+0000 7f1695aeab80 1 -- start start 2023-07-14T17:42:49.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.667+0000 7f1695aeab80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6af10 0x560918a6b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.667+0000 7f1695aeab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 0x560918a6be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.667+0000 7f1695aeab80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6c460 0x560918a6c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.667+0000 7f1695aeab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091896fc60 con 0x560918a6b860 2023-07-14T17:42:49.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.667+0000 7f1695aeab80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609189c29f0 con 0x560918a6af10 2023-07-14T17:42:49.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.667+0000 7f1695aeab80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609189bbe80 con 0x560918a6c460 2023-07-14T17:42:49.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.667+0000 7f16827fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 0x560918a6be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.667+0000 7f16827fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 0x560918a6be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41568/0 (socket says 172.21.15.88:41568) 2023-07-14T17:42:49.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.667+0000 7f16827fc700 1 -- 172.21.15.88:0/1743755891 learned_addr learned my addr 172.21.15.88:0/1743755891 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:49.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.667+0000 7f16837fe700 1 --2- 172.21.15.88:0/1743755891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6c460 0x560918a6c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.667+0000 7f1682ffd700 1 --2- 172.21.15.88:0/1743755891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6af10 0x560918a6b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.667+0000 7f16827fc700 1 -- 172.21.15.88:0/1743755891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6c460 msgr2=0x560918a6c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:42:49.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.667+0000 7f16827fc700 1 --2- 172.21.15.88:0/1743755891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6c460 0x560918a6c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.667+0000 7f16827fc700 1 -- 172.21.15.88:0/1743755891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6af10 msgr2=0x560918a6b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:42:49.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.667+0000 7f16827fc700 1 --2- 172.21.15.88:0/1743755891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6af10 0x560918a6b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.667+0000 7f16827fc700 1 -- 172.21.15.88:0/1743755891 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609187e9d90 con 0x560918a6b860 2023-07-14T17:42:49.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.668+0000 7f16827fc700 1 --2- 172.21.15.88:0/1743755891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 0x560918a6be80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f167800fd70 tx=0x7f1678012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c031712adb6951 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:49.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.668+0000 7f1669ffb700 1 -- 172.21.15.88:0/1743755891 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1678011c60 con 0x560918a6b860 2023-07-14T17:42:49.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.668+0000 7f1669ffb700 1 -- 172.21.15.88:0/1743755891 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1678011dc0 con 0x560918a6b860 2023-07-14T17:42:49.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.668+0000 7f1669ffb700 1 -- 172.21.15.88:0/1743755891 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167801d750 con 0x560918a6b860 2023-07-14T17:42:49.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.668+0000 7f1695aeab80 1 -- 172.21.15.88:0/1743755891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 msgr2=0x560918a6be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:49.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.668+0000 7f1695aeab80 1 --2- 172.21.15.88:0/1743755891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 0x560918a6be80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f167800fd70 tx=0x7f1678012000 comp rx=0 tx=0).stop 2023-07-14T17:42:49.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.668+0000 7f1695aeab80 1 -- 172.21.15.88:0/1743755891 shutdown_connections 2023-07-14T17:42:49.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.668+0000 7f1695aeab80 1 --2- 172.21.15.88:0/1743755891 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6c460 0x560918a6c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.668+0000 7f1695aeab80 1 --2- 172.21.15.88:0/1743755891 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6af10 0x560918a6b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.668+0000 7f1695aeab80 1 --2- 172.21.15.88:0/1743755891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 0x560918a6be80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.668+0000 7f1695aeab80 1 -- 172.21.15.88:0/1743755891 >> 172.21.15.88:0/1743755891 conn(0x560918a60630 msgr2=0x560918a65980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:49.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.668+0000 7f1695aeab80 1 -- 172.21.15.88:0/1743755891 shutdown_connections 2023-07-14T17:42:49.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.668+0000 7f1695aeab80 1 -- 172.21.15.88:0/1743755891 wait complete. 2023-07-14T17:42:49.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.669+0000 7f1695aeab80 1 -- start start 2023-07-14T17:42:49.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.669+0000 7f1695aeab80 1 Processor -- start 2023-07-14T17:42:49.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.669+0000 7f1695aeab80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6af10 0x560918a748f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.669+0000 7f1695aeab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 0x560918a74e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.669+0000 7f1695aeab80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6c460 0x560918a7be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.669+0000 7f1695aeab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091895ba70 con 0x560918a6b860 2023-07-14T17:42:49.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.669+0000 7f1695aeab80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091895b770 con 0x560918a6c460 2023-07-14T17:42:49.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.669+0000 7f1695aeab80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091883f8d0 con 0x560918a6af10 2023-07-14T17:42:49.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.669+0000 7f1695aeab80 10 client.0 _ll_register_callbacks cb 0x560918959710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:42:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.669+0000 7f16827fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 0x560918a74e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.669+0000 7f16827fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 0x560918a74e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41574/0 (socket says 172.21.15.88:41574) 2023-07-14T17:42:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.669+0000 7f16827fc700 1 -- 172.21.15.88:0/724414100 learned_addr learned my addr 172.21.15.88:0/724414100 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.669+0000 7f16837fe700 1 --2- 172.21.15.88:0/724414100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6c460 0x560918a7be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f1682ffd700 1 --2- 172.21.15.88:0/724414100 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6af10 0x560918a748f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f16827fc700 1 -- 172.21.15.88:0/724414100 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6af10 msgr2=0x560918a748f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:42:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f16827fc700 1 --2- 172.21.15.88:0/724414100 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6af10 0x560918a748f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f16827fc700 1 -- 172.21.15.88:0/724414100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6c460 msgr2=0x560918a7be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:42:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f16827fc700 1 --2- 172.21.15.88:0/724414100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6c460 0x560918a7be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f16827fc700 1 -- 172.21.15.88:0/724414100 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f167800fa20 con 0x560918a6b860 2023-07-14T17:42:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f16827fc700 1 --2- 172.21.15.88:0/724414100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 0x560918a74e30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f167800fd40 tx=0x7f1678005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e7d1a7294ecf49 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f1662ffd700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:42:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f1662ffd700 1 -- 172.21.15.88:0/724414100 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167802e070 con 0x560918a6b860 2023-07-14T17:42:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f1662ffd700 1 -- 172.21.15.88:0/724414100 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f167800fe10 con 0x560918a6b860 2023-07-14T17:42:49.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f1695aeab80 1 -- 172.21.15.88:0/724414100 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5609189154a0 con 0x560918a6b860 2023-07-14T17:42:49.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f1662ffd700 1 -- 172.21.15.88:0/724414100 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1678004710 con 0x560918a6b860 2023-07-14T17:42:49.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f1695aeab80 10 client.14745 fetch_fsmap learned FSMap version 17 2023-07-14T17:42:49.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f1695aeab80 1 -- 172.21.15.88:0/724414100 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x5609187d2c10 con 0x560918a6b860 2023-07-14T17:42:49.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f1662ffd700 1 -- 172.21.15.88:0/724414100 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x5609189154a0 con 0x560918a6b860 2023-07-14T17:42:49.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f1662ffd700 1 -- 172.21.15.88:0/724414100 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f167801e900 con 0x560918a6b860 2023-07-14T17:42:49.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f1695aeab80 10 client.14745 fetch_fsmap finished waiting for FSMap version 17 2023-07-14T17:42:49.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f1695aeab80 10 client.14745 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T17:42:49.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f1695aeab80 20 client.14745 populate_metadata read hostname 'smithi088' 2023-07-14T17:42:49.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f1695aeab80 1 --2- 172.21.15.88:0/724414100 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x560918a7fb30 0x560918a7eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f1695aeab80 4 client.14745 mds_command: new command op to 24506 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T17:42:49.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.670+0000 7f1695aeab80 1 -- 172.21.15.88:0/724414100 --> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560918a39d50 con 0x560918a7fb30 2023-07-14T17:42:49.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.671+0000 7f1682ffd700 1 --2- 172.21.15.88:0/724414100 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x560918a7fb30 0x560918a7eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.671+0000 7f1662ffd700 10 client.14745 ms_handle_connect on v2:172.21.15.107:6838/31646548 2023-07-14T17:42:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.671+0000 7f1682ffd700 1 --2- 172.21.15.88:0/724414100 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x560918a7fb30 0x560918a7eeb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44f0c38321a5da2e server_cookie=7697ed2e064adb30 in_seq=0 out_seq=0 2023-07-14T17:42:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.672+0000 7f1662ffd700 1 -- 172.21.15.88:0/724414100 <== mds.0 v2:172.21.15.107:6838/31646548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x560918a39d50 con 0x560918a7fb30 2023-07-14T17:42:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.672+0000 7f1662ffd700 10 client.14745 handle_command_reply: tid=0 2023-07-14T17:42:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.672+0000 7f1695aeab80 1 client.14745 shutdown 2023-07-14T17:42:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.672+0000 7f1695aeab80 1 -- 172.21.15.88:0/724414100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 msgr2=0x560918a74e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:49.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.672+0000 7f1695aeab80 1 --2- 172.21.15.88:0/724414100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 0x560918a74e30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f167800fd40 tx=0x7f1678005300 comp rx=0 tx=0).stop 2023-07-14T17:42:49.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.672+0000 7f1695aeab80 1 -- 172.21.15.88:0/724414100 shutdown_connections 2023-07-14T17:42:49.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.672+0000 7f1695aeab80 1 --2- 172.21.15.88:0/724414100 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x560918a7fb30 0x560918a7eeb0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.672+0000 7f1695aeab80 1 --2- 172.21.15.88:0/724414100 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6c460 0x560918a7be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.672+0000 7f1662ffd700 0 client.14745 ms_handle_reset on v2:172.21.15.107:6838/31646548 2023-07-14T17:42:49.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.672+0000 7f1695aeab80 1 --2- 172.21.15.88:0/724414100 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6af10 0x560918a748f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.672+0000 7f1695aeab80 1 --2- 172.21.15.88:0/724414100 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 0x560918a74e30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.672+0000 7f1695aeab80 1 -- 172.21.15.88:0/724414100 >> 172.21.15.88:0/724414100 conn(0x560918a60630 msgr2=0x560918a64600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:49.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.672+0000 7f1695aeab80 1 -- 172.21.15.88:0/724414100 shutdown_connections 2023-07-14T17:42:49.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.672+0000 7f1695aeab80 1 -- 172.21.15.88:0/724414100 wait complete. 2023-07-14T17:42:49.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.672+0000 7f1695aeab80 20 client.14745 trim_cache size 0 max 16384 2023-07-14T17:42:49.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f1695aeab80 1 Processor -- start 2023-07-14T17:42:49.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f1695aeab80 1 -- start start 2023-07-14T17:42:49.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f1695aeab80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6af10 0x560918a650a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f1695aeab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 0x560918a72780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f1695aeab80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6c460 0x560918a6f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f1695aeab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091898a7c0 con 0x560918a6b860 2023-07-14T17:42:49.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f1695aeab80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x56091884be10 con 0x560918a6af10 2023-07-14T17:42:49.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f1695aeab80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x560918957d50 con 0x560918a6c460 2023-07-14T17:42:49.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f16827fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 0x560918a72780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f16827fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 0x560918a72780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41582/0 (socket says 172.21.15.88:41582) 2023-07-14T17:42:49.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f16827fc700 1 -- 172.21.15.88:0/1310514424 learned_addr learned my addr 172.21.15.88:0/1310514424 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:49.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f16837fe700 1 --2- 172.21.15.88:0/1310514424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6c460 0x560918a6f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f1682ffd700 1 --2- 172.21.15.88:0/1310514424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6af10 0x560918a650a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f16827fc700 1 -- 172.21.15.88:0/1310514424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6c460 msgr2=0x560918a6f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:42:49.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f16827fc700 1 --2- 172.21.15.88:0/1310514424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6c460 0x560918a6f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f16827fc700 1 -- 172.21.15.88:0/1310514424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6af10 msgr2=0x560918a650a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:42:49.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f16827fc700 1 --2- 172.21.15.88:0/1310514424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6af10 0x560918a650a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f16827fc700 1 -- 172.21.15.88:0/1310514424 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560918a65f00 con 0x560918a6b860 2023-07-14T17:42:49.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f1682ffd700 1 --2- 172.21.15.88:0/1310514424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6af10 0x560918a650a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:42:49.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f16827fc700 1 --2- 172.21.15.88:0/1310514424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 0x560918a72780 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f167800fd10 tx=0x7f1678005420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07ad4da9e70bcff server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:49.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f166a7fc700 1 -- 172.21.15.88:0/1310514424 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167802e020 con 0x560918a6b860 2023-07-14T17:42:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f166a7fc700 1 -- 172.21.15.88:0/1310514424 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f16780279b0 con 0x560918a6b860 2023-07-14T17:42:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.690+0000 7f166a7fc700 1 -- 172.21.15.88:0/1310514424 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1678004710 con 0x560918a6b860 2023-07-14T17:42:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.691+0000 7f1695aeab80 1 -- 172.21.15.88:0/1310514424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 msgr2=0x560918a72780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.691+0000 7f1695aeab80 1 --2- 172.21.15.88:0/1310514424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 0x560918a72780 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f167800fd10 tx=0x7f1678005420 comp rx=0 tx=0).stop 2023-07-14T17:42:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.691+0000 7f1695aeab80 1 -- 172.21.15.88:0/1310514424 shutdown_connections 2023-07-14T17:42:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.691+0000 7f1695aeab80 1 --2- 172.21.15.88:0/1310514424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6c460 0x560918a6f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.691+0000 7f1695aeab80 1 --2- 172.21.15.88:0/1310514424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6af10 0x560918a650a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.691+0000 7f1695aeab80 1 --2- 172.21.15.88:0/1310514424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6b860 0x560918a72780 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.691+0000 7f1695aeab80 1 -- 172.21.15.88:0/1310514424 >> 172.21.15.88:0/1310514424 conn(0x560918a60630 msgr2=0x560918a736b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:49.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.691+0000 7f1695aeab80 1 -- 172.21.15.88:0/1310514424 shutdown_connections 2023-07-14T17:42:49.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.691+0000 7f1695aeab80 1 -- 172.21.15.88:0/1310514424 wait complete. 2023-07-14T17:42:49.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.691+0000 7f1695aeab80 1 -- start start 2023-07-14T17:42:49.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.691+0000 7f1695aeab80 1 Processor -- start 2023-07-14T17:42:49.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f1695aeab80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6af10 0x560918ad3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f1695aeab80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6b860 0x560918ad5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f1695aeab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6c460 0x560918ad8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f1695aeab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x560918ad8840 con 0x560918a6c460 2023-07-14T17:42:49.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f1695aeab80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x560918ad89b0 con 0x560918a6af10 2023-07-14T17:42:49.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f1695aeab80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x560918ad8b70 con 0x560918a6b860 2023-07-14T17:42:49.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f1695aeab80 10 client.0 _ll_register_callbacks cb 0x7f167800fc40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:42:49.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f16837fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6c460 0x560918ad8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f16837fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6c460 0x560918ad8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41590/0 (socket says 172.21.15.88:41590) 2023-07-14T17:42:49.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f16837fe700 1 -- 172.21.15.88:0/116235681 learned_addr learned my addr 172.21.15.88:0/116235681 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:49.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f16827fc700 1 --2- 172.21.15.88:0/116235681 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6b860 0x560918ad5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f16837fe700 1 -- 172.21.15.88:0/116235681 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6b860 msgr2=0x560918ad5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:42:49.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f1682ffd700 1 --2- 172.21.15.88:0/116235681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6af10 0x560918ad3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f16837fe700 1 --2- 172.21.15.88:0/116235681 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6b860 0x560918ad5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f16837fe700 1 -- 172.21.15.88:0/116235681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6af10 msgr2=0x560918ad3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:42:49.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f16837fe700 1 --2- 172.21.15.88:0/116235681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6af10 0x560918ad3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f16837fe700 1 -- 172.21.15.88:0/116235681 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5609187d2c10 con 0x560918a6c460 2023-07-14T17:42:49.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f16837fe700 1 --2- 172.21.15.88:0/116235681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6c460 0x560918ad8230 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f167c007b90 tx=0x7f167c007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6bb90eead65af40 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:49.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f1662ffd700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:42:49.698 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f1662ffd700 1 -- 172.21.15.88:0/116235681 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167c014d10 con 0x560918a6c460 2023-07-14T17:42:49.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f1662ffd700 1 -- 172.21.15.88:0/116235681 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f167c014e70 con 0x560918a6c460 2023-07-14T17:42:49.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f1695aeab80 1 -- 172.21.15.88:0/116235681 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1678011dc0 con 0x560918a6c460 2023-07-14T17:42:49.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f1662ffd700 1 -- 172.21.15.88:0/116235681 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f167c017440 con 0x560918a6c460 2023-07-14T17:42:49.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f1695aeab80 10 client.14751 fetch_fsmap learned FSMap version 17 2023-07-14T17:42:49.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.692+0000 7f1695aeab80 1 -- 172.21.15.88:0/116235681 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x560918810200 con 0x560918a6c460 2023-07-14T17:42:49.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.693+0000 7f1662ffd700 1 -- 172.21.15.88:0/116235681 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1678011dc0 con 0x560918a6c460 2023-07-14T17:42:49.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.693+0000 7f1662ffd700 1 -- 172.21.15.88:0/116235681 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f167c01f670 con 0x560918a6c460 2023-07-14T17:42:49.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.693+0000 7f1695aeab80 10 client.14751 fetch_fsmap finished waiting for FSMap version 17 2023-07-14T17:42:49.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.693+0000 7f1695aeab80 10 client.14751 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T17:42:49.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.693+0000 7f1695aeab80 20 client.14751 populate_metadata read hostname 'smithi088' 2023-07-14T17:42:49.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.693+0000 7f1695aeab80 1 --2- 172.21.15.88:0/116235681 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x560918add8d0 0x560918ada220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:49.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.693+0000 7f1695aeab80 4 client.14751 mds_command: new command op to 24506 tid=0[{"prefix": "status"}] 2023-07-14T17:42:49.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.693+0000 7f1695aeab80 1 -- 172.21.15.88:0/116235681 --> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5609189b4e60 con 0x560918add8d0 2023-07-14T17:42:49.700 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.693+0000 7f1682ffd700 1 --2- 172.21.15.88:0/116235681 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x560918add8d0 0x560918ada220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:49.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.693+0000 7f1662ffd700 10 client.14751 ms_handle_connect on v2:172.21.15.107:6838/31646548 2023-07-14T17:42:49.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.693+0000 7f1682ffd700 1 --2- 172.21.15.88:0/116235681 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x560918add8d0 0x560918ada220 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43370b6798c08df6 server_cookie=f16c4354afc37db3 in_seq=0 out_seq=0 2023-07-14T17:42:49.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1662ffd700 1 -- 172.21.15.88:0/116235681 <== mds.0 v2:172.21.15.107:6838/31646548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x5609189b4e60 con 0x560918add8d0 2023-07-14T17:42:49.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1662ffd700 10 client.14751 handle_command_reply: tid=0 2023-07-14T17:42:49.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1695aeab80 1 client.14751 shutdown 2023-07-14T17:42:49.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1695aeab80 1 -- 172.21.15.88:0/116235681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6c460 msgr2=0x560918ad8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:49.701 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1695aeab80 1 --2- 172.21.15.88:0/116235681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6c460 0x560918ad8230 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f167c007b90 tx=0x7f167c007f50 comp rx=0 tx=0).stop 2023-07-14T17:42:49.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1695aeab80 1 -- 172.21.15.88:0/116235681 shutdown_connections 2023-07-14T17:42:49.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1695aeab80 1 --2- 172.21.15.88:0/116235681 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x560918add8d0 0x560918ada220 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1695aeab80 1 --2- 172.21.15.88:0/116235681 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x560918a6c460 0x560918ad8230 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1662ffd700 0 client.14751 ms_handle_reset on v2:172.21.15.107:6838/31646548 2023-07-14T17:42:49.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1695aeab80 1 --2- 172.21.15.88:0/116235681 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x560918a6af10 0x560918ad3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1695aeab80 1 --2- 172.21.15.88:0/116235681 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x560918a6b860 0x560918ad5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1695aeab80 1 -- 172.21.15.88:0/116235681 >> 172.21.15.88:0/116235681 conn(0x560918a60630 msgr2=0x560918a650a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:49.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1695aeab80 1 -- 172.21.15.88:0/116235681 shutdown_connections 2023-07-14T17:42:49.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1695aeab80 1 -- 172.21.15.88:0/116235681 wait complete. 2023-07-14T17:42:49.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1695aeab80 20 client.14751 trim_cache size 0 max 16384 2023-07-14T17:42:49.703 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-07-14T17:42:49.704 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "b462a556-226c-11ee-9b33-001a4aab830c", 2023-07-14T17:42:49.704 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2023-07-14T17:42:49.704 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 24506, 2023-07-14T17:42:49.704 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2023-07-14T17:42:49.704 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2023-07-14T17:42:49.704 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2023-07-14T17:42:49.704 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 6.7798812899999996, 2023-07-14T17:42:49.705 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 17, 2023-07-14T17:42:49.705 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 86, 2023-07-14T17:42:49.705 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 86, 2023-07-14T17:42:49.705 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 11.866225644 2023-07-14T17:42:49.705 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-07-14T17:42:49.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1689952700 1 -- 172.21.15.88:0/2282327594 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f166c0a1150 msgr2=0x7f166c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:49.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1689952700 1 --2- 172.21.15.88:0/2282327594 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f166c0a1150 0x7f166c0a3600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1678005950 tx=0x7f16780058e0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1689952700 1 -- 172.21.15.88:0/2282327594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1684100770 msgr2=0x7f1684195870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:49.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1689952700 1 --2- 172.21.15.88:0/2282327594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1684100770 0x7f1684195870 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f16740098e0 tx=0x7f16740039d0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1689952700 1 -- 172.21.15.88:0/2282327594 shutdown_connections 2023-07-14T17:42:49.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1689952700 1 --2- 172.21.15.88:0/2282327594 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f16841059b0 0x7f168419ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.694+0000 7f1689952700 1 --2- 172.21.15.88:0/2282327594 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f166c0a1150 0x7f166c0a3600 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.695+0000 7f1689952700 1 --2- 172.21.15.88:0/2282327594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1684100770 0x7f1684195870 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.695+0000 7f1689952700 1 --2- 172.21.15.88:0/2282327594 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1684103090 0x7f1684195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:49.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.695+0000 7f1689952700 1 -- 172.21.15.88:0/2282327594 >> 172.21.15.88:0/2282327594 conn(0x7f16840fa150 msgr2=0x7f16841013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:49.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.695+0000 7f1689952700 1 -- 172.21.15.88:0/2282327594 shutdown_connections 2023-07-14T17:42:49.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:49.695+0000 7f1689952700 1 -- 172.21.15.88:0/2282327594 wait complete. 2023-07-14T17:42:49.709 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-07-14T17:42:49.709 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-07-14T17:42:49.710 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph fs ls 2023-07-14T17:42:49.941 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:49 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3561451689' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:49.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:49 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2896787877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:49.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:49 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1247638051' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:49.974 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:49 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3561451689' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:49.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:49 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2896787877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:49.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:49 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1247638051' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:50.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:49 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3561451689' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:50.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:49 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2896787877' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:42:50.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:49 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1247638051' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:50.255 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:42:50.799 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:50 smithi088 ceph-mon[127465]: pgmap v259: 129 pgs: 8 creating+peering, 4 unknown, 117 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2023-07-14T17:42:50.974 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:50 smithi107 ceph-mon[128999]: pgmap v259: 129 pgs: 8 creating+peering, 4 unknown, 117 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2023-07-14T17:42:51.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:50 smithi178 ceph-mon[129368]: pgmap v259: 129 pgs: 8 creating+peering, 4 unknown, 117 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2023-07-14T17:42:51.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.446+0000 7f6fa250f700 1 -- 172.21.15.88:0/310882550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f9c10a2d0 msgr2=0x7f6f9c10c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:51.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.446+0000 7f6fa250f700 1 --2- 172.21.15.88:0/310882550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f9c10a2d0 0x7f6f9c10c6f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6f9400b0a0 tx=0x7f6f9400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:51.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.446+0000 7f6fa250f700 1 -- 172.21.15.88:0/310882550 shutdown_connections 2023-07-14T17:42:51.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.446+0000 7f6fa250f700 1 --2- 172.21.15.88:0/310882550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f9c10a2d0 0x7f6f9c10c6f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:51.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.446+0000 7f6fa250f700 1 --2- 172.21.15.88:0/310882550 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f9c100ef0 0x7f6f9c1012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:51.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.446+0000 7f6fa250f700 1 --2- 172.21.15.88:0/310882550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f9c101800 0x7f6f9c109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:51.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.446+0000 7f6fa250f700 1 -- 172.21.15.88:0/310882550 >> 172.21.15.88:0/310882550 conn(0x7f6f9c075d20 msgr2=0x7f6f9c076120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:51.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.446+0000 7f6fa250f700 1 -- 172.21.15.88:0/310882550 shutdown_connections 2023-07-14T17:42:51.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.446+0000 7f6fa250f700 1 -- 172.21.15.88:0/310882550 wait complete. 2023-07-14T17:42:51.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.447+0000 7f6fa250f700 1 Processor -- start 2023-07-14T17:42:51.431 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.447+0000 7f6fa250f700 1 -- start start 2023-07-14T17:42:51.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.447+0000 7f6fa250f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f9c100ef0 0x7f6f9c0feb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:51.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.447+0000 7f6fa250f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f9c101800 0x7f6f9c0ff090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:51.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.447+0000 7f6fa250f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f9c10a2d0 0x7f6f9c0ff610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:51.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.447+0000 7f6fa250f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c1a4240 con 0x7f6f9c10a2d0 2023-07-14T17:42:51.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.447+0000 7f6fa250f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c1a43b0 con 0x7f6f9c101800 2023-07-14T17:42:51.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.447+0000 7f6fa250f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c1a4570 con 0x7f6f9c100ef0 2023-07-14T17:42:51.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.447+0000 7f6fa0aac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f9c10a2d0 0x7f6f9c0ff610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:51.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.447+0000 7f6fa0aac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f9c10a2d0 0x7f6f9c0ff610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41604/0 (socket says 172.21.15.88:41604) 2023-07-14T17:42:51.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.447+0000 7f6fa0aac700 1 -- 172.21.15.88:0/50430855 learned_addr learned my addr 172.21.15.88:0/50430855 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:51.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.447+0000 7f6f9b7fe700 1 --2- 172.21.15.88:0/50430855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f9c101800 0x7f6f9c0ff090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:51.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.447+0000 7f6fa0aac700 1 -- 172.21.15.88:0/50430855 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f9c100ef0 msgr2=0x7f6f9c0feb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:51.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.447+0000 7f6fa0aac700 1 --2- 172.21.15.88:0/50430855 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f9c100ef0 0x7f6f9c0feb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:51.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.447+0000 7f6fa0aac700 1 -- 172.21.15.88:0/50430855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f9c101800 msgr2=0x7f6f9c0ff090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:51.434 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.447+0000 7f6fa0aac700 1 --2- 172.21.15.88:0/50430855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f9c101800 0x7f6f9c0ff090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.447+0000 7f6fa0aac700 1 -- 172.21.15.88:0/50430855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f94009d00 con 0x7f6f9c10a2d0 2023-07-14T17:42:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.447+0000 7f6f9b7fe700 1 --2- 172.21.15.88:0/50430855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f9c101800 0x7f6f9c0ff090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:42:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.448+0000 7f6fa0aac700 1 --2- 172.21.15.88:0/50430855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f9c10a2d0 0x7f6f9c0ff610 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6f9400b070 tx=0x7f6f94004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.448+0000 7f6f997fa700 1 -- 172.21.15.88:0/50430855 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f9400e040 con 0x7f6f9c10a2d0 2023-07-14T17:42:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.448+0000 7f6fa250f700 1 -- 172.21.15.88:0/50430855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f9c1a47f0 con 0x7f6f9c10a2d0 2023-07-14T17:42:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.448+0000 7f6f997fa700 1 -- 172.21.15.88:0/50430855 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6f94003e90 con 0x7f6f9c10a2d0 2023-07-14T17:42:51.435 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.448+0000 7f6f997fa700 1 -- 172.21.15.88:0/50430855 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f94004020 con 0x7f6f9c10a2d0 2023-07-14T17:42:51.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.448+0000 7f6fa250f700 1 -- 172.21.15.88:0/50430855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f9c1a4cf0 con 0x7f6f9c10a2d0 2023-07-14T17:42:51.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.449+0000 7f6fa250f700 1 -- 172.21.15.88:0/50430855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9c04e340 con 0x7f6f9c10a2d0 2023-07-14T17:42:51.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.450+0000 7f6f997fa700 1 -- 172.21.15.88:0/50430855 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6f9401b6f0 con 0x7f6f9c10a2d0 2023-07-14T17:42:51.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.450+0000 7f6f997fa700 1 --2- 172.21.15.88:0/50430855 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6f880a1150 0x7f6f880a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:51.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.450+0000 7f6f9bfff700 1 --2- 172.21.15.88:0/50430855 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6f880a1150 0x7f6f880a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:51.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.450+0000 7f6f997fa700 1 -- 172.21.15.88:0/50430855 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6f940ce8c0 con 0x7f6f9c10a2d0 2023-07-14T17:42:51.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.451+0000 7f6f9bfff700 1 --2- 172.21.15.88:0/50430855 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6f880a1150 0x7f6f880a3600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6f840097b0 tx=0x7f6f84006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:51.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.454+0000 7f6f997fa700 1 -- 172.21.15.88:0/50430855 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6f94097eb0 con 0x7f6f9c10a2d0 2023-07-14T17:42:51.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.681+0000 7f6fa250f700 1 -- 172.21.15.88:0/50430855 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6f9c100080 con 0x7f6f9c10a2d0 2023-07-14T17:42:51.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.681+0000 7f6f997fa700 1 -- 172.21.15.88:0/50430855 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7f6f94009500 con 0x7f6f9c10a2d0 2023-07-14T17:42:51.665 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-07-14T17:42:51.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.683+0000 7f6fa250f700 1 -- 172.21.15.88:0/50430855 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6f880a1150 msgr2=0x7f6f880a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:51.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.683+0000 7f6fa250f700 1 --2- 172.21.15.88:0/50430855 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6f880a1150 0x7f6f880a3600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f6f840097b0 tx=0x7f6f84006d20 comp rx=0 tx=0).stop 2023-07-14T17:42:51.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.683+0000 7f6fa250f700 1 -- 172.21.15.88:0/50430855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f9c10a2d0 msgr2=0x7f6f9c0ff610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:51.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.683+0000 7f6fa250f700 1 --2- 172.21.15.88:0/50430855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f9c10a2d0 0x7f6f9c0ff610 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6f9400b070 tx=0x7f6f94004db0 comp rx=0 tx=0).stop 2023-07-14T17:42:51.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.683+0000 7f6fa250f700 1 -- 172.21.15.88:0/50430855 shutdown_connections 2023-07-14T17:42:51.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.683+0000 7f6fa250f700 1 --2- 172.21.15.88:0/50430855 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6f880a1150 0x7f6f880a3600 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:51.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.683+0000 7f6fa250f700 1 --2- 172.21.15.88:0/50430855 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f9c10a2d0 0x7f6f9c0ff610 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:51.667 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.683+0000 7f6fa250f700 1 --2- 172.21.15.88:0/50430855 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6f9c100ef0 0x7f6f9c0feb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:51.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.683+0000 7f6fa250f700 1 --2- 172.21.15.88:0/50430855 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6f9c101800 0x7f6f9c0ff090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:51.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.683+0000 7f6fa250f700 1 -- 172.21.15.88:0/50430855 >> 172.21.15.88:0/50430855 conn(0x7f6f9c075d20 msgr2=0x7f6f9c0fd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:51.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.683+0000 7f6fa250f700 1 -- 172.21.15.88:0/50430855 shutdown_connections 2023-07-14T17:42:51.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:51.683+0000 7f6fa250f700 1 -- 172.21.15.88:0/50430855 wait complete. 2023-07-14T17:42:51.974 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:51 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/50430855' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T17:42:52.029 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:51 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/50430855' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T17:42:52.097 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-07-14T17:42:52.098 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-14T17:42:52.098 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2023-07-14T17:42:52.098 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-07-14T17:42:52.098 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-07-14T17:42:52.099 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-07-14T17:42:52.099 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-07-14T17:42:52.099 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-14T17:42:52.099 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-07-14T17:42:52.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:42:52.100 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2023-07-14T17:42:52.118 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-14T17:42:52.118 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-14T17:42:52.118 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2023-07-14T17:42:52.118 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2023-07-14T17:42:52.118 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2023-07-14T17:42:52.118 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2023-07-14T17:42:52.119 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T17:42:52.119 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2023-07-14T17:42:52.119 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T17:42:52.119 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2023-07-14T17:42:52.119 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T17:42:52.119 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2023-07-14T17:42:52.119 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-14T17:42:52.119 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2023-07-14T17:42:52.120 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-07-14T17:42:52.120 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 41520sec preferred_lft 41520sec 2023-07-14T17:42:52.120 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::4893:55f2:c841:20/64 scope link noprefixroute 2023-07-14T17:42:52.120 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2023-07-14T17:42:52.121 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-07-14T17:42:52.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:42:52.121 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-07-14T17:42:52.122 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-07-14T17:42:52.122 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link add name ceph-brx type bridge 2023-07-14T17:42:52.122 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip addr flush dev ceph-brx 2023-07-14T17:42:52.122 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set ceph-brx up 2023-07-14T17:42:52.123 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-07-14T17:42:52.123 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-07-14T17:42:52.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:51 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/50430855' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T17:42:52.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:42:52.293 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-14T17:42:52.320 INFO:teuthology.orchestra.run.smithi088.stdout:1 2023-07-14T17:42:52.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:42:52.322 DEBUG:teuthology.orchestra.run.smithi088:> route 2023-07-14T17:42:52.393 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2023-07-14T17:42:52.394 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-14T17:42:52.394 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-07-14T17:42:52.394 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-07-14T17:42:52.394 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-14T17:42:52.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:42:52.395 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-07-14T17:42:52.395 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-07-14T17:42:52.395 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-07-14T17:42:52.396 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-07-14T17:42:52.396 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-07-14T17:42:52.396 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-07-14T17:42:52.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:42:52.639 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2023-07-14T17:42:52.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:42:52.656 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list-id 2023-07-14T17:42:52.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:42:52.713 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-07-14T17:42:52.713 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-07-14T17:42:52.713 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-07-14T17:42:52.714 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-07-14T17:42:52.714 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-07-14T17:42:52.806 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-07-14T17:42:52.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:42:52.807 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-07-14T17:42:52.807 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-07-14T17:42:52.808 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-07-14T17:42:52.808 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-07-14T17:42:52.808 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-07-14T17:42:52.809 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-07-14T17:42:52.809 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-07-14T17:42:52.809 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-07-14T17:42:52.847 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:52 smithi088 ceph-mon[127465]: pgmap v260: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2023-07-14T17:42:53.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:42:53.013 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-07-14T17:42:53.013 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-07-14T17:42:53.013 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2023-07-14T17:42:53.014 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set dev brx.0 master ceph-brx 2023-07-14T17:42:53.014 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-07-14T17:42:53.150 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-07-14T17:42:53.151 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T17:42:53.151 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-07-14T17:42:53.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:52 smithi178 ceph-mon[129368]: pgmap v260: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2023-07-14T17:42:53.168 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-07-14T17:42:53.169 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T17:42:53.169 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-07-14T17:42:53.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:52 smithi107 ceph-mon[128999]: pgmap v260: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 1 op/s 2023-07-14T17:42:53.225 DEBUG:teuthology.orchestra.run.smithi088:> sudo modprobe fuse 2023-07-14T17:42:53.292 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-14T17:42:53.293 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-14T17:42:53.359 INFO:teuthology.orchestra.run.smithi088.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-14T17:42:53.361 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-14T17:42:53.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:42:53.362 DEBUG:teuthology.orchestra.run.smithi088:> ls /sys/fs/fuse/connections 2023-07-14T17:42:53.418 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-07-14T17:42:53.419 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-07-14T17:42:53.461 DEBUG:teuthology.orchestra.run.smithi088:> sudo modprobe fuse 2023-07-14T17:42:53.487 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-14T17:42:53.487 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-14T17:42:53.552 INFO:teuthology.orchestra.run.smithi088.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-14T17:42:53.553 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-14T17:42:53.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:42:53.554 DEBUG:teuthology.orchestra.run.smithi088:> ls /sys/fs/fuse/connections 2023-07-14T17:42:53.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi088.stderr:2023-07-14T17:42:53.597+0000 7f1fb0e095c0 -1 init, newargv = 0x563483fc4c80 newargc=15 2023-07-14T17:42:53.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi088.stderr:ceph-fuse[164682]: starting ceph client 2023-07-14T17:42:53.587 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi088.stderr:ceph-fuse[164682]: starting fuse 2023-07-14T17:42:53.595 INFO:teuthology.orchestra.run.smithi088.stdout:124 2023-07-14T17:42:53.596 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [124] 2023-07-14T17:42:53.596 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-07-14T17:42:53.597 DEBUG:teuthology.orchestra.run.smithi088:> import glob 2023-07-14T17:42:53.597 DEBUG:teuthology.orchestra.run.smithi088:> import re 2023-07-14T17:42:53.597 DEBUG:teuthology.orchestra.run.smithi088:> import os 2023-07-14T17:42:53.597 DEBUG:teuthology.orchestra.run.smithi088:> import subprocess 2023-07-14T17:42:53.598 DEBUG:teuthology.orchestra.run.smithi088:> 2023-07-14T17:42:53.598 DEBUG:teuthology.orchestra.run.smithi088:> def _find_admin_socket(client_name): 2023-07-14T17:42:53.598 DEBUG:teuthology.orchestra.run.smithi088:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-07-14T17:42:53.599 DEBUG:teuthology.orchestra.run.smithi088:> files = glob.glob(asok_path) 2023-07-14T17:42:53.599 DEBUG:teuthology.orchestra.run.smithi088:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-07-14T17:42:53.599 DEBUG:teuthology.orchestra.run.smithi088:> 2023-07-14T17:42:53.599 DEBUG:teuthology.orchestra.run.smithi088:> # Given a non-glob path, it better be there 2023-07-14T17:42:53.600 DEBUG:teuthology.orchestra.run.smithi088:> if "*" not in asok_path: 2023-07-14T17:42:53.600 DEBUG:teuthology.orchestra.run.smithi088:> assert(len(files) == 1) 2023-07-14T17:42:53.600 DEBUG:teuthology.orchestra.run.smithi088:> return files[0] 2023-07-14T17:42:53.600 DEBUG:teuthology.orchestra.run.smithi088:> 2023-07-14T17:42:53.601 DEBUG:teuthology.orchestra.run.smithi088:> for f in files: 2023-07-14T17:42:53.601 DEBUG:teuthology.orchestra.run.smithi088:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-07-14T17:42:53.601 DEBUG:teuthology.orchestra.run.smithi088:> if os.path.exists("/proc/{0}".format(pid)): 2023-07-14T17:42:53.602 DEBUG:teuthology.orchestra.run.smithi088:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-07-14T17:42:53.602 DEBUG:teuthology.orchestra.run.smithi088:> contents = proc_f.read() 2023-07-14T17:42:53.602 DEBUG:teuthology.orchestra.run.smithi088:> if mountpoint in contents: 2023-07-14T17:42:53.602 DEBUG:teuthology.orchestra.run.smithi088:> return f 2023-07-14T17:42:53.603 DEBUG:teuthology.orchestra.run.smithi088:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-07-14T17:42:53.603 DEBUG:teuthology.orchestra.run.smithi088:> 2023-07-14T17:42:53.603 DEBUG:teuthology.orchestra.run.smithi088:> print(_find_admin_socket("client.admin")) 2023-07-14T17:42:53.603 DEBUG:teuthology.orchestra.run.smithi088:> ' 2023-07-14T17:42:53.719 INFO:teuthology.orchestra.run.smithi088.stdout:/var/run/ceph/ceph-client.admin.164682.asok 2023-07-14T17:42:53.901 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.164682.asok 2023-07-14T17:42:53.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:42:53.902 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.164682.asok status 2023-07-14T17:42:54.058 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-07-14T17:42:54.058 INFO:teuthology.orchestra.run.smithi088.stdout: "metadata": { 2023-07-14T17:42:54.058 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_sha1": "81c9387d2e7576da065747ec9c16d1c7d69016ec", 2023-07-14T17:42:54.059 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T17:42:54.059 INFO:teuthology.orchestra.run.smithi088.stdout: "entity_id": "admin", 2023-07-14T17:42:54.059 INFO:teuthology.orchestra.run.smithi088.stdout: "hostname": "smithi088", 2023-07-14T17:42:54.059 INFO:teuthology.orchestra.run.smithi088.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-07-14T17:42:54.059 INFO:teuthology.orchestra.run.smithi088.stdout: "pid": "164682", 2023-07-14T17:42:54.059 INFO:teuthology.orchestra.run.smithi088.stdout: "root": "/" 2023-07-14T17:42:54.059 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-07-14T17:42:54.060 INFO:teuthology.orchestra.run.smithi088.stdout: "dentry_count": 0, 2023-07-14T17:42:54.060 INFO:teuthology.orchestra.run.smithi088.stdout: "dentry_pinned_count": 0, 2023-07-14T17:42:54.060 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 14763, 2023-07-14T17:42:54.060 INFO:teuthology.orchestra.run.smithi088.stdout: "inst": { 2023-07-14T17:42:54.060 INFO:teuthology.orchestra.run.smithi088.stdout: "name": { 2023-07-14T17:42:54.060 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "client", 2023-07-14T17:42:54.060 INFO:teuthology.orchestra.run.smithi088.stdout: "num": 14763 2023-07-14T17:42:54.061 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-07-14T17:42:54.061 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": { 2023-07-14T17:42:54.061 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2023-07-14T17:42:54.061 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "192.168.0.1:0", 2023-07-14T17:42:54.061 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3891890671 2023-07-14T17:42:54.061 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-07-14T17:42:54.061 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-07-14T17:42:54.061 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": { 2023-07-14T17:42:54.062 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2023-07-14T17:42:54.062 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "192.168.0.1:0", 2023-07-14T17:42:54.062 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 3891890671 2023-07-14T17:42:54.062 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-07-14T17:42:54.062 INFO:teuthology.orchestra.run.smithi088.stdout: "inst_str": "client.14763 192.168.0.1:0/3891890671", 2023-07-14T17:42:54.062 INFO:teuthology.orchestra.run.smithi088.stdout: "addr_str": "192.168.0.1:0/3891890671", 2023-07-14T17:42:54.062 INFO:teuthology.orchestra.run.smithi088.stdout: "inode_count": 1, 2023-07-14T17:42:54.063 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_epoch": 17, 2023-07-14T17:42:54.063 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_epoch": 86, 2023-07-14T17:42:54.063 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_epoch_barrier": 0, 2023-07-14T17:42:54.063 INFO:teuthology.orchestra.run.smithi088.stdout: "blocklisted": false, 2023-07-14T17:42:54.063 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs" 2023-07-14T17:42:54.063 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-07-14T17:42:54.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:42:54.068 DEBUG:teuthology.orchestra.run.smithi088:> stat --file-system '--printf=%T 2023-07-14T17:42:54.069 DEBUG:teuthology.orchestra.run.smithi088:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-07-14T17:42:54.086 INFO:teuthology.orchestra.run.smithi088.stdout:fuseblk 2023-07-14T17:42:54.086 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-07-14T17:42:54.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:42:54.087 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-07-14T17:42:54.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:42:54.154 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-07-14T17:42:54.260 INFO:teuthology.orchestra.run.smithi088.stdout:# file: . 2023-07-14T17:42:54.260 INFO:teuthology.orchestra.run.smithi088.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-07-14T17:42:54.260 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:54.261 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:42:54.261 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T17:42:54.278 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:42:54.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.455+0000 7f0596657700 1 -- 172.21.15.88:0/1044713356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0590100e20 msgr2=0x7f0590101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:54.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.455+0000 7f0596657700 1 --2- 172.21.15.88:0/1044713356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0590100e20 0x7f0590101270 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0584009990 tx=0x7f0584009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.455+0000 7f0596657700 1 -- 172.21.15.88:0/1044713356 shutdown_connections 2023-07-14T17:42:54.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.455+0000 7f0596657700 1 --2- 172.21.15.88:0/1044713356 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05901017b0 0x7f0590108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.442 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.455+0000 7f0596657700 1 --2- 172.21.15.88:0/1044713356 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05900ffc30 0x7f0590100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.455+0000 7f0596657700 1 --2- 172.21.15.88:0/1044713356 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0590100e20 0x7f0590101270 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.455+0000 7f0596657700 1 -- 172.21.15.88:0/1044713356 >> 172.21.15.88:0/1044713356 conn(0x7f05900fb230 msgr2=0x7f05900fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:54.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.455+0000 7f0596657700 1 -- 172.21.15.88:0/1044713356 shutdown_connections 2023-07-14T17:42:54.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.455+0000 7f0596657700 1 -- 172.21.15.88:0/1044713356 wait complete. 2023-07-14T17:42:54.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.456+0000 7f0596657700 1 Processor -- start 2023-07-14T17:42:54.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.456+0000 7f0596657700 1 -- start start 2023-07-14T17:42:54.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.456+0000 7f0596657700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05900ffc30 0x7f05901958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:54.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.456+0000 7f0596657700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0590100e20 0x7f0590195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:54.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.456+0000 7f0596657700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05901017b0 0x7f059019ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:54.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.456+0000 7f0596657700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059019d430 con 0x7f05901017b0 2023-07-14T17:42:54.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.456+0000 7f0596657700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059019d570 con 0x7f0590100e20 2023-07-14T17:42:54.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.456+0000 7f0596657700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059019d700 con 0x7f05900ffc30 2023-07-14T17:42:54.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.456+0000 7f0594bf4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05901017b0 0x7f059019ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:54.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.456+0000 7f0594bf4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05901017b0 0x7f059019ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41632/0 (socket says 172.21.15.88:41632) 2023-07-14T17:42:54.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.456+0000 7f0594bf4700 1 -- 172.21.15.88:0/442873683 learned_addr learned my addr 172.21.15.88:0/442873683 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:54.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.457+0000 7f058f7fe700 1 --2- 172.21.15.88:0/442873683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0590100e20 0x7f0590195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:54.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.457+0000 7f058ffff700 1 --2- 172.21.15.88:0/442873683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05900ffc30 0x7f05901958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:54.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.457+0000 7f0594bf4700 1 -- 172.21.15.88:0/442873683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05900ffc30 msgr2=0x7f05901958b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:54.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.457+0000 7f0594bf4700 1 --2- 172.21.15.88:0/442873683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05900ffc30 0x7f05901958b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.457+0000 7f0594bf4700 1 -- 172.21.15.88:0/442873683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0590100e20 msgr2=0x7f0590195df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:54.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.457+0000 7f0594bf4700 1 --2- 172.21.15.88:0/442873683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0590100e20 0x7f0590195df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.457+0000 7f0594bf4700 1 -- 172.21.15.88:0/442873683 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05840095f0 con 0x7f05901017b0 2023-07-14T17:42:54.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.457+0000 7f0594bf4700 1 --2- 172.21.15.88:0/442873683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05901017b0 0x7f059019ce60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0588007ee0 tx=0x7f058800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:54.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.457+0000 7f058d7fa700 1 -- 172.21.15.88:0/442873683 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588017070 con 0x7f05901017b0 2023-07-14T17:42:54.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.457+0000 7f0596657700 1 -- 172.21.15.88:0/442873683 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f059019d9e0 con 0x7f05901017b0 2023-07-14T17:42:54.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.457+0000 7f058d7fa700 1 -- 172.21.15.88:0/442873683 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f058800f040 con 0x7f05901017b0 2023-07-14T17:42:54.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.457+0000 7f058d7fa700 1 -- 172.21.15.88:0/442873683 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588005710 con 0x7f05901017b0 2023-07-14T17:42:54.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.457+0000 7f0596657700 1 -- 172.21.15.88:0/442873683 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f059019df20 con 0x7f05901017b0 2023-07-14T17:42:54.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.458+0000 7f0596657700 1 -- 172.21.15.88:0/442873683 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059010b350 con 0x7f05901017b0 2023-07-14T17:42:54.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.459+0000 7f058d7fa700 1 -- 172.21.15.88:0/442873683 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0588005870 con 0x7f05901017b0 2023-07-14T17:42:54.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.460+0000 7f058d7fa700 1 --2- 172.21.15.88:0/442873683 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f05780a11a0 0x7f05780a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:54.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.460+0000 7f058ffff700 1 --2- 172.21.15.88:0/442873683 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f05780a11a0 0x7f05780a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:54.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.460+0000 7f058d7fa700 1 -- 172.21.15.88:0/442873683 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f05880cc9f0 con 0x7f05901017b0 2023-07-14T17:42:54.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.460+0000 7f058ffff700 1 --2- 172.21.15.88:0/442873683 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f05780a11a0 0x7f05780a3650 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0580006fd0 tx=0x7f0580008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:54.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.464+0000 7f058d7fa700 1 -- 172.21.15.88:0/442873683 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0588095f50 con 0x7f05901017b0 2023-07-14T17:42:54.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.657+0000 7f0596657700 1 -- 172.21.15.88:0/442873683 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0590106b50 con 0x7f05901017b0 2023-07-14T17:42:54.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.658+0000 7f058d7fa700 1 -- 172.21.15.88:0/442873683 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23985 (secure 0 0 0) 0x7f0588099570 con 0x7f05901017b0 2023-07-14T17:42:54.642 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:54.642 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":86,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:42:46.934364+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:42:54.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.660+0000 7f0596657700 1 -- 172.21.15.88:0/442873683 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f05780a11a0 msgr2=0x7f05780a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:54.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.660+0000 7f0596657700 1 --2- 172.21.15.88:0/442873683 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f05780a11a0 0x7f05780a3650 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0580006fd0 tx=0x7f0580008040 comp rx=0 tx=0).stop 2023-07-14T17:42:54.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.660+0000 7f0596657700 1 -- 172.21.15.88:0/442873683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05901017b0 msgr2=0x7f059019ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:54.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.660+0000 7f0596657700 1 --2- 172.21.15.88:0/442873683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05901017b0 0x7f059019ce60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0588007ee0 tx=0x7f058800d3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.660+0000 7f0596657700 1 -- 172.21.15.88:0/442873683 shutdown_connections 2023-07-14T17:42:54.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.660+0000 7f0596657700 1 --2- 172.21.15.88:0/442873683 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f05780a11a0 0x7f05780a3650 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.660+0000 7f0596657700 1 --2- 172.21.15.88:0/442873683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05901017b0 0x7f059019ce60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.660+0000 7f0596657700 1 --2- 172.21.15.88:0/442873683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05900ffc30 0x7f05901958b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.660+0000 7f0596657700 1 --2- 172.21.15.88:0/442873683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0590100e20 0x7f0590195df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.660+0000 7f0596657700 1 -- 172.21.15.88:0/442873683 >> 172.21.15.88:0/442873683 conn(0x7f05900fb230 msgr2=0x7f05900fd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:54.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.660+0000 7f0596657700 1 -- 172.21.15.88:0/442873683 shutdown_connections 2023-07-14T17:42:54.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.660+0000 7f0596657700 1 -- 172.21.15.88:0/442873683 wait complete. 2023-07-14T17:42:54.656 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T17:38:50.126535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T17:42:40.725870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T17:42:41.512749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T17:42:44.764573+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T17:42:54.657 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T17:42:54.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.797+0000 7f2045ba4700 1 -- 172.21.15.88:0/994971672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20400feb30 msgr2=0x7f20400fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:54.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.797+0000 7f2045ba4700 1 --2- 172.21.15.88:0/994971672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20400feb30 0x7f20400fef40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f2034009990 tx=0x7f2034009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.797+0000 7f2045ba4700 1 -- 172.21.15.88:0/994971672 shutdown_connections 2023-07-14T17:42:54.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.797+0000 7f2045ba4700 1 --2- 172.21.15.88:0/994971672 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20401006b0 0x7f2040106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.797+0000 7f2045ba4700 1 --2- 172.21.15.88:0/994971672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20400feb30 0x7f20400fef40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.797+0000 7f2045ba4700 1 --2- 172.21.15.88:0/994971672 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20400ffd20 0x7f2040100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.797+0000 7f2045ba4700 1 -- 172.21.15.88:0/994971672 >> 172.21.15.88:0/994971672 conn(0x7f20400fa190 msgr2=0x7f20400fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:54.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.797+0000 7f2045ba4700 1 -- 172.21.15.88:0/994971672 shutdown_connections 2023-07-14T17:42:54.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.797+0000 7f2045ba4700 1 -- 172.21.15.88:0/994971672 wait complete. 2023-07-14T17:42:54.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.798+0000 7f2045ba4700 1 Processor -- start 2023-07-14T17:42:54.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.798+0000 7f2045ba4700 1 -- start start 2023-07-14T17:42:54.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.798+0000 7f2045ba4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20400feb30 0x7f2040197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:54.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.798+0000 7f2045ba4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20400ffd20 0x7f2040197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:54.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.798+0000 7f2045ba4700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20401006b0 0x7f204019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:54.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.798+0000 7f2045ba4700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204019f630 con 0x7f20400ffd20 2023-07-14T17:42:54.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.798+0000 7f2045ba4700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204019f770 con 0x7f20400feb30 2023-07-14T17:42:54.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.798+0000 7f2045ba4700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204019f900 con 0x7f20401006b0 2023-07-14T17:42:54.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.798+0000 7f203effd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20400ffd20 0x7f2040197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:54.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.798+0000 7f203effd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20400ffd20 0x7f2040197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41654/0 (socket says 172.21.15.88:41654) 2023-07-14T17:42:54.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.798+0000 7f203effd700 1 -- 172.21.15.88:0/2760118388 learned_addr learned my addr 172.21.15.88:0/2760118388 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:54.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.798+0000 7f203f7fe700 1 --2- 172.21.15.88:0/2760118388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20400feb30 0x7f2040197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:54.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.798+0000 7f203ffff700 1 --2- 172.21.15.88:0/2760118388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20401006b0 0x7f204019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:54.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.798+0000 7f203effd700 1 -- 172.21.15.88:0/2760118388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20401006b0 msgr2=0x7f204019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:54.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.798+0000 7f203effd700 1 --2- 172.21.15.88:0/2760118388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20401006b0 0x7f204019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.798+0000 7f203effd700 1 -- 172.21.15.88:0/2760118388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20400feb30 msgr2=0x7f2040197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:54.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.798+0000 7f203effd700 1 --2- 172.21.15.88:0/2760118388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20400feb30 0x7f2040197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.798+0000 7f203effd700 1 -- 172.21.15.88:0/2760118388 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20340095f0 con 0x7f20400ffd20 2023-07-14T17:42:54.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.799+0000 7f203effd700 1 --2- 172.21.15.88:0/2760118388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20400ffd20 0x7f2040197ff0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f203000c920 tx=0x7f203000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:54.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.799+0000 7f203cff9700 1 -- 172.21.15.88:0/2760118388 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2030007b00 con 0x7f20400ffd20 2023-07-14T17:42:54.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.799+0000 7f203cff9700 1 -- 172.21.15.88:0/2760118388 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f203000ce70 con 0x7f20400ffd20 2023-07-14T17:42:54.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.799+0000 7f2045ba4700 1 -- 172.21.15.88:0/2760118388 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204019fbe0 con 0x7f20400ffd20 2023-07-14T17:42:54.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.799+0000 7f203cff9700 1 -- 172.21.15.88:0/2760118388 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2030005440 con 0x7f20400ffd20 2023-07-14T17:42:54.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.799+0000 7f2045ba4700 1 -- 172.21.15.88:0/2760118388 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20401a00f0 con 0x7f20400ffd20 2023-07-14T17:42:54.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.800+0000 7f2045ba4700 1 -- 172.21.15.88:0/2760118388 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f204010a1b0 con 0x7f20400ffd20 2023-07-14T17:42:54.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.802+0000 7f203cff9700 1 -- 172.21.15.88:0/2760118388 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f20300055a0 con 0x7f20400ffd20 2023-07-14T17:42:54.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.802+0000 7f203cff9700 1 --2- 172.21.15.88:0/2760118388 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f20280a1150 0x7f20280a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:54.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.802+0000 7f203f7fe700 1 --2- 172.21.15.88:0/2760118388 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f20280a1150 0x7f20280a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:54.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.802+0000 7f203cff9700 1 -- 172.21.15.88:0/2760118388 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f20300ccad0 con 0x7f20400ffd20 2023-07-14T17:42:54.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.803+0000 7f203f7fe700 1 --2- 172.21.15.88:0/2760118388 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f20280a1150 0x7f20280a3600 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2034006010 tx=0x7f203400b2a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:54.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.806+0000 7f203cff9700 1 -- 172.21.15.88:0/2760118388 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2030096030 con 0x7f20400ffd20 2023-07-14T17:42:54.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.993+0000 7f2045ba4700 1 -- 172.21.15.88:0/2760118388 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f20401a06a0 con 0x7f20400ffd20 2023-07-14T17:42:54.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.993+0000 7f203cff9700 1 -- 172.21.15.88:0/2760118388 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f203001d020 con 0x7f20400ffd20 2023-07-14T17:42:54.977 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-07-14T17:42:54.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.995+0000 7f2045ba4700 1 -- 172.21.15.88:0/2760118388 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f20280a1150 msgr2=0x7f20280a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:54.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.995+0000 7f2045ba4700 1 --2- 172.21.15.88:0/2760118388 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f20280a1150 0x7f20280a3600 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2034006010 tx=0x7f203400b2a0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.995+0000 7f2045ba4700 1 -- 172.21.15.88:0/2760118388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20400ffd20 msgr2=0x7f2040197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:54.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.995+0000 7f2045ba4700 1 --2- 172.21.15.88:0/2760118388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20400ffd20 0x7f2040197ff0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f203000c920 tx=0x7f203000cc30 comp rx=0 tx=0).stop 2023-07-14T17:42:54.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.995+0000 7f2045ba4700 1 -- 172.21.15.88:0/2760118388 shutdown_connections 2023-07-14T17:42:54.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.995+0000 7f2045ba4700 1 --2- 172.21.15.88:0/2760118388 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20401006b0 0x7f204019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.995+0000 7f2045ba4700 1 --2- 172.21.15.88:0/2760118388 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20400feb30 0x7f2040197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.995+0000 7f2045ba4700 1 --2- 172.21.15.88:0/2760118388 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f20280a1150 0x7f20280a3600 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.995+0000 7f2045ba4700 1 --2- 172.21.15.88:0/2760118388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20400ffd20 0x7f2040197ff0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:54.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.995+0000 7f2045ba4700 1 -- 172.21.15.88:0/2760118388 >> 172.21.15.88:0/2760118388 conn(0x7f20400fa190 msgr2=0x7f2040068a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:54.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.995+0000 7f2045ba4700 1 -- 172.21.15.88:0/2760118388 shutdown_connections 2023-07-14T17:42:54.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:54.996+0000 7f2045ba4700 1 -- 172.21.15.88:0/2760118388 wait complete. 2023-07-14T17:42:54.990 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T17:42:55.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.131+0000 7fef16693700 1 -- 172.21.15.88:0/2209365654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef100fed00 msgr2=0x7fef100ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:55.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.131+0000 7fef16693700 1 --2- 172.21.15.88:0/2209365654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef100fed00 0x7fef100ff170 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fef00009990 tx=0x7fef00009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.131+0000 7fef16693700 1 -- 172.21.15.88:0/2209365654 shutdown_connections 2023-07-14T17:42:55.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.131+0000 7fef16693700 1 --2- 172.21.15.88:0/2209365654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef100ff6b0 0x7fef10106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.131+0000 7fef16693700 1 --2- 172.21.15.88:0/2209365654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef101000b0 0x7fef101004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.131+0000 7fef16693700 1 --2- 172.21.15.88:0/2209365654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef100fed00 0x7fef100ff170 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.131+0000 7fef16693700 1 -- 172.21.15.88:0/2209365654 >> 172.21.15.88:0/2209365654 conn(0x7fef100f98a0 msgr2=0x7fef100fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:55.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.131+0000 7fef16693700 1 -- 172.21.15.88:0/2209365654 shutdown_connections 2023-07-14T17:42:55.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.131+0000 7fef16693700 1 -- 172.21.15.88:0/2209365654 wait complete. 2023-07-14T17:42:55.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.131+0000 7fef16693700 1 Processor -- start 2023-07-14T17:42:55.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.131+0000 7fef16693700 1 -- start start 2023-07-14T17:42:55.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.131+0000 7fef16693700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef100fed00 0x7fef100fe780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:55.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef16693700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef100ff6b0 0x7fef1019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:55.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef16693700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef101000b0 0x7fef1019cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:55.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef16693700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1019d3e0 con 0x7fef100ff6b0 2023-07-14T17:42:55.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef16693700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1019d550 con 0x7fef101000b0 2023-07-14T17:42:55.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef16693700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1019d710 con 0x7fef100fed00 2023-07-14T17:42:55.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef0f7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef100ff6b0 0x7fef1019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:55.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef0f7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef100ff6b0 0x7fef1019a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41668/0 (socket says 172.21.15.88:41668) 2023-07-14T17:42:55.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef0f7fe700 1 -- 172.21.15.88:0/2703329281 learned_addr learned my addr 172.21.15.88:0/2703329281 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:55.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef14c30700 1 --2- 172.21.15.88:0/2703329281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef101000b0 0x7fef1019cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:55.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef0ffff700 1 --2- 172.21.15.88:0/2703329281 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef100fed00 0x7fef100fe780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:55.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef0f7fe700 1 -- 172.21.15.88:0/2703329281 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef100fed00 msgr2=0x7fef100fe780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:55.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef0f7fe700 1 --2- 172.21.15.88:0/2703329281 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef100fed00 0x7fef100fe780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef0f7fe700 1 -- 172.21.15.88:0/2703329281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef101000b0 msgr2=0x7fef1019cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:55.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef0f7fe700 1 --2- 172.21.15.88:0/2703329281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef101000b0 0x7fef1019cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef0f7fe700 1 -- 172.21.15.88:0/2703329281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef000095f0 con 0x7fef100ff6b0 2023-07-14T17:42:55.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef14c30700 1 --2- 172.21.15.88:0/2703329281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef101000b0 0x7fef1019cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:42:55.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef0f7fe700 1 --2- 172.21.15.88:0/2703329281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef100ff6b0 0x7fef1019a5c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fef00006010 tx=0x7fef00004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:55.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef0d7fa700 1 -- 172.21.15.88:0/2703329281 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0001d070 con 0x7fef100ff6b0 2023-07-14T17:42:55.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef16693700 1 -- 172.21.15.88:0/2703329281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef1019d990 con 0x7fef100ff6b0 2023-07-14T17:42:55.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef0d7fa700 1 -- 172.21.15.88:0/2703329281 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fef00004080 con 0x7fef100ff6b0 2023-07-14T17:42:55.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef0d7fa700 1 -- 172.21.15.88:0/2703329281 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef00022c90 con 0x7fef100ff6b0 2023-07-14T17:42:55.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.132+0000 7fef16693700 1 -- 172.21.15.88:0/2703329281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef1019de10 con 0x7fef100ff6b0 2023-07-14T17:42:55.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.133+0000 7fef16693700 1 -- 172.21.15.88:0/2703329281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef1010a160 con 0x7fef100ff6b0 2023-07-14T17:42:55.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.134+0000 7fef0d7fa700 1 -- 172.21.15.88:0/2703329281 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fef00022470 con 0x7fef100ff6b0 2023-07-14T17:42:55.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.135+0000 7fef0d7fa700 1 --2- 172.21.15.88:0/2703329281 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7feef80a1150 0x7feef80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:55.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.135+0000 7fef0ffff700 1 --2- 172.21.15.88:0/2703329281 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7feef80a1150 0x7feef80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:55.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.135+0000 7fef0d7fa700 1 -- 172.21.15.88:0/2703329281 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fef000ce1f0 con 0x7fef100ff6b0 2023-07-14T17:42:55.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.135+0000 7fef0ffff700 1 --2- 172.21.15.88:0/2703329281 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7feef80a1150 0x7feef80a3600 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fef04005950 tx=0x7fef0400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:55.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.138+0000 7fef0d7fa700 1 -- 172.21.15.88:0/2703329281 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fef000977e0 con 0x7fef100ff6b0 2023-07-14T17:42:55.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:54 smithi178 ceph-mon[129368]: pgmap v261: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-14T17:42:55.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:54 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/442873683' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:55.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:54 smithi088 ceph-mon[127465]: pgmap v261: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-14T17:42:55.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:54 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/442873683' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:55.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:54 smithi107 ceph-mon[128999]: pgmap v261: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-14T17:42:55.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:54 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/442873683' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:55.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.330+0000 7fef16693700 1 -- 172.21.15.88:0/2703329281 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fef1010a320 con 0x7fef100ff6b0 2023-07-14T17:42:55.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.330+0000 7fef0d7fa700 1 -- 172.21.15.88:0/2703329281 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) v1 ==== 104+0+11 (secure 0 0 0) 0x7fef0009ae00 con 0x7fef100ff6b0 2023-07-14T17:42:55.314 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-07-14T17:42:55.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.332+0000 7fef16693700 1 -- 172.21.15.88:0/2703329281 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7feef80a1150 msgr2=0x7feef80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:55.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.332+0000 7fef16693700 1 --2- 172.21.15.88:0/2703329281 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7feef80a1150 0x7feef80a3600 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fef04005950 tx=0x7fef0400b410 comp rx=0 tx=0).stop 2023-07-14T17:42:55.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.332+0000 7fef16693700 1 -- 172.21.15.88:0/2703329281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef100ff6b0 msgr2=0x7fef1019a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:55.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.332+0000 7fef16693700 1 --2- 172.21.15.88:0/2703329281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef100ff6b0 0x7fef1019a5c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fef00006010 tx=0x7fef00004730 comp rx=0 tx=0).stop 2023-07-14T17:42:55.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.332+0000 7fef16693700 1 -- 172.21.15.88:0/2703329281 shutdown_connections 2023-07-14T17:42:55.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.332+0000 7fef16693700 1 --2- 172.21.15.88:0/2703329281 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fef101000b0 0x7fef1019cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.332+0000 7fef16693700 1 --2- 172.21.15.88:0/2703329281 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fef100fed00 0x7fef100fe780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.332+0000 7fef16693700 1 --2- 172.21.15.88:0/2703329281 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7feef80a1150 0x7feef80a3600 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.332+0000 7fef16693700 1 --2- 172.21.15.88:0/2703329281 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef100ff6b0 0x7fef1019a5c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.332+0000 7fef16693700 1 -- 172.21.15.88:0/2703329281 >> 172.21.15.88:0/2703329281 conn(0x7fef100f98a0 msgr2=0x7fef100fbc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:55.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.332+0000 7fef16693700 1 -- 172.21.15.88:0/2703329281 shutdown_connections 2023-07-14T17:42:55.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.332+0000 7fef16693700 1 -- 172.21.15.88:0/2703329281 wait complete. 2023-07-14T17:42:55.327 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T17:42:55.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.468+0000 7effe9668700 1 -- 172.21.15.88:0/1099812097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effe40ffe40 msgr2=0x7effe4106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:55.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.468+0000 7effe9668700 1 --2- 172.21.15.88:0/1099812097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effe40ffe40 0x7effe4106700 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7effdc00c210 tx=0x7effdc00c520 comp rx=0 tx=0).stop 2023-07-14T17:42:55.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.468+0000 7effe9668700 1 -- 172.21.15.88:0/1099812097 shutdown_connections 2023-07-14T17:42:55.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.468+0000 7effe9668700 1 --2- 172.21.15.88:0/1099812097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effe40ffe40 0x7effe4106700 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.468+0000 7effe9668700 1 --2- 172.21.15.88:0/1099812097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effe40fe2c0 0x7effe40fe6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.468+0000 7effe9668700 1 --2- 172.21.15.88:0/1099812097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7effe40ff4b0 0x7effe40ff900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.468+0000 7effe9668700 1 -- 172.21.15.88:0/1099812097 >> 172.21.15.88:0/1099812097 conn(0x7effe40f98e0 msgr2=0x7effe40fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:55.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.468+0000 7effe9668700 1 -- 172.21.15.88:0/1099812097 shutdown_connections 2023-07-14T17:42:55.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.468+0000 7effe9668700 1 -- 172.21.15.88:0/1099812097 wait complete. 2023-07-14T17:42:55.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.468+0000 7effe9668700 1 Processor -- start 2023-07-14T17:42:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.469+0000 7effe9668700 1 -- start start 2023-07-14T17:42:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.469+0000 7effe9668700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effe40fe2c0 0x7effe41959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.469+0000 7effe9668700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7effe40ff4b0 0x7effe4195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.469+0000 7effe9668700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effe40ffe40 0x7effe419cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.469+0000 7effe9668700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe419d540 con 0x7effe40fe2c0 2023-07-14T17:42:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.469+0000 7effe9668700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe419d680 con 0x7effe40ffe40 2023-07-14T17:42:55.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.469+0000 7effe9668700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe419d810 con 0x7effe40ff4b0 2023-07-14T17:42:55.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.469+0000 7effe2ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effe40fe2c0 0x7effe41959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:55.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.469+0000 7effe2ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effe40fe2c0 0x7effe41959c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41680/0 (socket says 172.21.15.88:41680) 2023-07-14T17:42:55.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.469+0000 7effe2ffd700 1 -- 172.21.15.88:0/1328095980 learned_addr learned my addr 172.21.15.88:0/1328095980 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:55.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.469+0000 7effe27fc700 1 --2- 172.21.15.88:0/1328095980 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7effe40ff4b0 0x7effe4195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:55.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.469+0000 7effe37fe700 1 --2- 172.21.15.88:0/1328095980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effe40ffe40 0x7effe419cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:55.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.469+0000 7effe2ffd700 1 -- 172.21.15.88:0/1328095980 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7effe40ff4b0 msgr2=0x7effe4195f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:55.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.469+0000 7effe2ffd700 1 --2- 172.21.15.88:0/1328095980 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7effe40ff4b0 0x7effe4195f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.469+0000 7effe2ffd700 1 -- 172.21.15.88:0/1328095980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effe40ffe40 msgr2=0x7effe419cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:55.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.469+0000 7effe2ffd700 1 --2- 172.21.15.88:0/1328095980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effe40ffe40 0x7effe419cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.469+0000 7effe2ffd700 1 -- 172.21.15.88:0/1328095980 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc00ae10 con 0x7effe40fe2c0 2023-07-14T17:42:55.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.469+0000 7effe2ffd700 1 --2- 172.21.15.88:0/1328095980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effe40fe2c0 0x7effe41959c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7effd400e970 tx=0x7effd400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:55.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.470+0000 7effdbfff700 1 -- 172.21.15.88:0/1328095980 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4009790 con 0x7effe40fe2c0 2023-07-14T17:42:55.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.470+0000 7effdbfff700 1 -- 172.21.15.88:0/1328095980 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7effd400eec0 con 0x7effe40fe2c0 2023-07-14T17:42:55.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.470+0000 7effe9668700 1 -- 172.21.15.88:0/1328095980 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe419daf0 con 0x7effe40fe2c0 2023-07-14T17:42:55.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.470+0000 7effdbfff700 1 -- 172.21.15.88:0/1328095980 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4003c10 con 0x7effe40fe2c0 2023-07-14T17:42:55.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.470+0000 7effe9668700 1 -- 172.21.15.88:0/1328095980 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe419e050 con 0x7effe40fe2c0 2023-07-14T17:42:55.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.471+0000 7effe9668700 1 -- 172.21.15.88:0/1328095980 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe4109190 con 0x7effe40fe2c0 2023-07-14T17:42:55.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.472+0000 7effdbfff700 1 -- 172.21.15.88:0/1328095980 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7effd4003db0 con 0x7effe40fe2c0 2023-07-14T17:42:55.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.472+0000 7effdbfff700 1 --2- 172.21.15.88:0/1328095980 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7effd00a11d0 0x7effd00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:55.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.472+0000 7effe27fc700 1 --2- 172.21.15.88:0/1328095980 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7effd00a11d0 0x7effd00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:55.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.472+0000 7effdbfff700 1 -- 172.21.15.88:0/1328095980 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7effd4014070 con 0x7effe40fe2c0 2023-07-14T17:42:55.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.473+0000 7effe27fc700 1 --2- 172.21.15.88:0/1328095980 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7effd00a11d0 0x7effd00a3680 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7effcc009c60 tx=0x7effcc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:55.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.476+0000 7effdbfff700 1 -- 172.21.15.88:0/1328095980 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7effd4096260 con 0x7effe40fe2c0 2023-07-14T17:42:55.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.669+0000 7effe9668700 1 -- 172.21.15.88:0/1328095980 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7effe4105150 con 0x7effe40fe2c0 2023-07-14T17:42:55.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.671+0000 7effdbfff700 1 -- 172.21.15.88:0/1328095980 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) v1 ==== 100+0+11 (secure 0 0 0) 0x7effd4099880 con 0x7effe40fe2c0 2023-07-14T17:42:55.654 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 64 2023-07-14T17:42:55.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.672+0000 7effe9668700 1 -- 172.21.15.88:0/1328095980 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7effd00a11d0 msgr2=0x7effd00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:55.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.672+0000 7effe9668700 1 --2- 172.21.15.88:0/1328095980 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7effd00a11d0 0x7effd00a3680 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7effcc009c60 tx=0x7effcc009260 comp rx=0 tx=0).stop 2023-07-14T17:42:55.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.672+0000 7effe9668700 1 -- 172.21.15.88:0/1328095980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effe40fe2c0 msgr2=0x7effe41959c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:55.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.672+0000 7effe9668700 1 --2- 172.21.15.88:0/1328095980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effe40fe2c0 0x7effe41959c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7effd400e970 tx=0x7effd400ec80 comp rx=0 tx=0).stop 2023-07-14T17:42:55.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.672+0000 7effe9668700 1 -- 172.21.15.88:0/1328095980 shutdown_connections 2023-07-14T17:42:55.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.672+0000 7effe9668700 1 --2- 172.21.15.88:0/1328095980 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7effe40ffe40 0x7effe419cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.672+0000 7effe9668700 1 --2- 172.21.15.88:0/1328095980 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7effd00a11d0 0x7effd00a3680 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.672+0000 7effe9668700 1 --2- 172.21.15.88:0/1328095980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7effe40fe2c0 0x7effe41959c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.672+0000 7effe9668700 1 --2- 172.21.15.88:0/1328095980 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7effe40ff4b0 0x7effe4195f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.672+0000 7effe9668700 1 -- 172.21.15.88:0/1328095980 >> 172.21.15.88:0/1328095980 conn(0x7effe40f98e0 msgr2=0x7effe40fbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:55.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.672+0000 7effe9668700 1 -- 172.21.15.88:0/1328095980 shutdown_connections 2023-07-14T17:42:55.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.672+0000 7effe9668700 1 -- 172.21.15.88:0/1328095980 wait complete. 2023-07-14T17:42:55.667 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T17:42:55.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.810+0000 7f6140570700 1 -- 172.21.15.88:0/4223452856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61380ffd30 msgr2=0x7f6138100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:55.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.810+0000 7f6140570700 1 --2- 172.21.15.88:0/4223452856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61380ffd30 0x7f6138100180 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6134009990 tx=0x7f6134009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.810+0000 7f6140570700 1 -- 172.21.15.88:0/4223452856 shutdown_connections 2023-07-14T17:42:55.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.810+0000 7f6140570700 1 --2- 172.21.15.88:0/4223452856 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61381006c0 0x7f6138106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.810+0000 7f6140570700 1 --2- 172.21.15.88:0/4223452856 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61380feb40 0x7f61380fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.810+0000 7f6140570700 1 --2- 172.21.15.88:0/4223452856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61380ffd30 0x7f6138100180 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.810+0000 7f6140570700 1 -- 172.21.15.88:0/4223452856 >> 172.21.15.88:0/4223452856 conn(0x7f61380fa160 msgr2=0x7f61380fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:55.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.810+0000 7f6140570700 1 -- 172.21.15.88:0/4223452856 shutdown_connections 2023-07-14T17:42:55.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.810+0000 7f6140570700 1 -- 172.21.15.88:0/4223452856 wait complete. 2023-07-14T17:42:55.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f6140570700 1 Processor -- start 2023-07-14T17:42:55.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f6140570700 1 -- start start 2023-07-14T17:42:55.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f6140570700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61380feb40 0x7f6138194790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:55.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f6140570700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61380ffd30 0x7f6138194cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:55.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f6140570700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61381006c0 0x7f613819bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:55.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f6140570700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613819c310 con 0x7f61380ffd30 2023-07-14T17:42:55.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f6140570700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613819c450 con 0x7f61380feb40 2023-07-14T17:42:55.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f6140570700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613819c5e0 con 0x7f61381006c0 2023-07-14T17:42:55.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f613db0b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61380ffd30 0x7f6138194cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:55.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f613db0b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61380ffd30 0x7f6138194cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41696/0 (socket says 172.21.15.88:41696) 2023-07-14T17:42:55.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f613db0b700 1 -- 172.21.15.88:0/1646255181 learned_addr learned my addr 172.21.15.88:0/1646255181 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:55.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f613e30c700 1 --2- 172.21.15.88:0/1646255181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61380feb40 0x7f6138194790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:55.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f613eb0d700 1 --2- 172.21.15.88:0/1646255181 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61381006c0 0x7f613819bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:55.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f613db0b700 1 -- 172.21.15.88:0/1646255181 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61381006c0 msgr2=0x7f613819bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:55.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f613db0b700 1 --2- 172.21.15.88:0/1646255181 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61381006c0 0x7f613819bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f613db0b700 1 -- 172.21.15.88:0/1646255181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61380feb40 msgr2=0x7f6138194790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:55.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f613db0b700 1 --2- 172.21.15.88:0/1646255181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61380feb40 0x7f6138194790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f613db0b700 1 -- 172.21.15.88:0/1646255181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61340095f0 con 0x7f61380ffd30 2023-07-14T17:42:55.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f613eb0d700 1 --2- 172.21.15.88:0/1646255181 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61381006c0 0x7f613819bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:42:55.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f613e30c700 1 --2- 172.21.15.88:0/1646255181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61380feb40 0x7f6138194790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:42:55.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f613db0b700 1 --2- 172.21.15.88:0/1646255181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61380ffd30 0x7f6138194cd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6134003a50 tx=0x7f6134009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:55.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.811+0000 7f612b7fe700 1 -- 172.21.15.88:0/1646255181 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613401d070 con 0x7f61380ffd30 2023-07-14T17:42:55.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.812+0000 7f612b7fe700 1 -- 172.21.15.88:0/1646255181 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6134005a00 con 0x7f61380ffd30 2023-07-14T17:42:55.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.812+0000 7f6140570700 1 -- 172.21.15.88:0/1646255181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f613819c860 con 0x7f61380ffd30 2023-07-14T17:42:55.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.812+0000 7f612b7fe700 1 -- 172.21.15.88:0/1646255181 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6134005280 con 0x7f61380ffd30 2023-07-14T17:42:55.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.812+0000 7f6140570700 1 -- 172.21.15.88:0/1646255181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f613819ccb0 con 0x7f61380ffd30 2023-07-14T17:42:55.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.813+0000 7f6140570700 1 -- 172.21.15.88:0/1646255181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f613810a1c0 con 0x7f61380ffd30 2023-07-14T17:42:55.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.814+0000 7f612b7fe700 1 -- 172.21.15.88:0/1646255181 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f61340053e0 con 0x7f61380ffd30 2023-07-14T17:42:55.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.814+0000 7f612b7fe700 1 --2- 172.21.15.88:0/1646255181 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f61240a1150 0x7f61240a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:55.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.815+0000 7f613e30c700 1 --2- 172.21.15.88:0/1646255181 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f61240a1150 0x7f61240a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:55.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.815+0000 7f612b7fe700 1 -- 172.21.15.88:0/1646255181 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f61340ce150 con 0x7f61380ffd30 2023-07-14T17:42:55.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.815+0000 7f613e30c700 1 --2- 172.21.15.88:0/1646255181 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f61240a1150 0x7f61240a3600 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f612c005950 tx=0x7f612c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:55.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:55.818+0000 7f612b7fe700 1 -- 172.21.15.88:0/1646255181 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6134097740 con 0x7f61380ffd30 2023-07-14T17:42:55.941 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:55 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2760118388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:42:55.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:55 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2703329281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:42:55.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:55 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1328095980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:42:55.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.010+0000 7f6140570700 1 -- 172.21.15.88:0/1646255181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6138105ae0 con 0x7f61380ffd30 2023-07-14T17:42:55.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.010+0000 7f612b7fe700 1 -- 172.21.15.88:0/1646255181 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v86) v1 ==== 103+0+11 (secure 0 0 0) 0x7f613409ad60 con 0x7f61380ffd30 2023-07-14T17:42:55.993 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-07-14T17:42:55.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.012+0000 7f6140570700 1 -- 172.21.15.88:0/1646255181 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f61240a1150 msgr2=0x7f61240a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:55.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.012+0000 7f6140570700 1 --2- 172.21.15.88:0/1646255181 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f61240a1150 0x7f61240a3600 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f612c005950 tx=0x7f612c0058e0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.012+0000 7f6140570700 1 -- 172.21.15.88:0/1646255181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61380ffd30 msgr2=0x7f6138194cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:55.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.012+0000 7f6140570700 1 --2- 172.21.15.88:0/1646255181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61380ffd30 0x7f6138194cd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6134003a50 tx=0x7f6134009580 comp rx=0 tx=0).stop 2023-07-14T17:42:55.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.013+0000 7f6140570700 1 -- 172.21.15.88:0/1646255181 shutdown_connections 2023-07-14T17:42:55.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.013+0000 7f6140570700 1 --2- 172.21.15.88:0/1646255181 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f61381006c0 0x7f613819bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.013+0000 7f6140570700 1 --2- 172.21.15.88:0/1646255181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f61380feb40 0x7f6138194790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.013+0000 7f6140570700 1 --2- 172.21.15.88:0/1646255181 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f61240a1150 0x7f61240a3600 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.013+0000 7f6140570700 1 --2- 172.21.15.88:0/1646255181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61380ffd30 0x7f6138194cd0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:55.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.013+0000 7f6140570700 1 -- 172.21.15.88:0/1646255181 >> 172.21.15.88:0/1646255181 conn(0x7f61380fa160 msgr2=0x7f61380fc410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:55.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.013+0000 7f6140570700 1 -- 172.21.15.88:0/1646255181 shutdown_connections 2023-07-14T17:42:55.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.013+0000 7f6140570700 1 -- 172.21.15.88:0/1646255181 wait complete. 2023-07-14T17:42:56.007 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T17:42:56.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.147+0000 7fc8646ad700 1 -- 172.21.15.88:0/3885573524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc85c0feb40 msgr2=0x7fc85c0fef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:56.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.147+0000 7fc8646ad700 1 --2- 172.21.15.88:0/3885573524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc85c0feb40 0x7fc85c0fef50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc858009990 tx=0x7fc858009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.147+0000 7fc8646ad700 1 -- 172.21.15.88:0/3885573524 shutdown_connections 2023-07-14T17:42:56.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.147+0000 7fc8646ad700 1 --2- 172.21.15.88:0/3885573524 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc85c1006c0 0x7fc85c106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.147+0000 7fc8646ad700 1 --2- 172.21.15.88:0/3885573524 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc85c0feb40 0x7fc85c0fef50 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.147+0000 7fc8646ad700 1 --2- 172.21.15.88:0/3885573524 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc85c0ffd30 0x7fc85c100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.147+0000 7fc8646ad700 1 -- 172.21.15.88:0/3885573524 >> 172.21.15.88:0/3885573524 conn(0x7fc85c0fa160 msgr2=0x7fc85c0fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:56.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.147+0000 7fc8646ad700 1 -- 172.21.15.88:0/3885573524 shutdown_connections 2023-07-14T17:42:56.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.147+0000 7fc8646ad700 1 -- 172.21.15.88:0/3885573524 wait complete. 2023-07-14T17:42:56.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.147+0000 7fc8646ad700 1 Processor -- start 2023-07-14T17:42:56.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.147+0000 7fc8646ad700 1 -- start start 2023-07-14T17:42:56.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.147+0000 7fc8646ad700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc85c0feb40 0x7fc85c069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:56.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.147+0000 7fc8646ad700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc85c0ffd30 0x7fc85c19a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:56.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.147+0000 7fc8646ad700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc85c1006c0 0x7fc85c19cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:56.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.147+0000 7fc8646ad700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c19d3f0 con 0x7fc85c0ffd30 2023-07-14T17:42:56.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.147+0000 7fc8646ad700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c19d560 con 0x7fc85c0feb40 2023-07-14T17:42:56.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.147+0000 7fc8646ad700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85c19d720 con 0x7fc85c1006c0 2023-07-14T17:42:56.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.148+0000 7fc861c48700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc85c0ffd30 0x7fc85c19a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.148+0000 7fc861c48700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc85c0ffd30 0x7fc85c19a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41710/0 (socket says 172.21.15.88:41710) 2023-07-14T17:42:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.148+0000 7fc861c48700 1 -- 172.21.15.88:0/3107938854 learned_addr learned my addr 172.21.15.88:0/3107938854 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.148+0000 7fc862449700 1 --2- 172.21.15.88:0/3107938854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc85c0feb40 0x7fc85c069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.148+0000 7fc862c4a700 1 --2- 172.21.15.88:0/3107938854 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc85c1006c0 0x7fc85c19cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.148+0000 7fc861c48700 1 -- 172.21.15.88:0/3107938854 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc85c1006c0 msgr2=0x7fc85c19cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.148+0000 7fc861c48700 1 --2- 172.21.15.88:0/3107938854 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc85c1006c0 0x7fc85c19cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.148+0000 7fc861c48700 1 -- 172.21.15.88:0/3107938854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc85c0feb40 msgr2=0x7fc85c069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:56.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.148+0000 7fc861c48700 1 --2- 172.21.15.88:0/3107938854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc85c0feb40 0x7fc85c069340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.148+0000 7fc861c48700 1 -- 172.21.15.88:0/3107938854 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8580095f0 con 0x7fc85c0ffd30 2023-07-14T17:42:56.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.148+0000 7fc861c48700 1 --2- 172.21.15.88:0/3107938854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc85c0ffd30 0x7fc85c19a5c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00b530 tx=0x7fc84c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:56.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.148+0000 7fc8537fe700 1 -- 172.21.15.88:0/3107938854 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c011830 con 0x7fc85c0ffd30 2023-07-14T17:42:56.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.148+0000 7fc8537fe700 1 -- 172.21.15.88:0/3107938854 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc84c011e20 con 0x7fc85c0ffd30 2023-07-14T17:42:56.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.148+0000 7fc8646ad700 1 -- 172.21.15.88:0/3107938854 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc85c19da00 con 0x7fc85c0ffd30 2023-07-14T17:42:56.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.148+0000 7fc8537fe700 1 -- 172.21.15.88:0/3107938854 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c010dd0 con 0x7fc85c0ffd30 2023-07-14T17:42:56.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.148+0000 7fc8646ad700 1 -- 172.21.15.88:0/3107938854 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85c19df90 con 0x7fc85c0ffd30 2023-07-14T17:42:56.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.149+0000 7fc8646ad700 1 -- 172.21.15.88:0/3107938854 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc85c10a240 con 0x7fc85c0ffd30 2023-07-14T17:42:56.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.150+0000 7fc8537fe700 1 -- 172.21.15.88:0/3107938854 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc84c011990 con 0x7fc85c0ffd30 2023-07-14T17:42:56.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.151+0000 7fc8537fe700 1 --2- 172.21.15.88:0/3107938854 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc8480a1150 0x7fc8480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:56.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.151+0000 7fc8537fe700 1 -- 172.21.15.88:0/3107938854 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc84c0cca70 con 0x7fc85c0ffd30 2023-07-14T17:42:56.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.151+0000 7fc862449700 1 --2- 172.21.15.88:0/3107938854 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc8480a1150 0x7fc8480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:56.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.151+0000 7fc862449700 1 --2- 172.21.15.88:0/3107938854 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc8480a1150 0x7fc8480a3600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc85800c010 tx=0x7fc85800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:56.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.154+0000 7fc8537fe700 1 -- 172.21.15.88:0/3107938854 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc84c095fd0 con 0x7fc85c0ffd30 2023-07-14T17:42:56.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:55 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2760118388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:42:56.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:55 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2703329281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:42:56.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:55 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1328095980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:42:56.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:55 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2760118388' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:42:56.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:55 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2703329281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:42:56.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:55 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1328095980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:42:56.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.359+0000 7fc8646ad700 1 -- 172.21.15.88:0/3107938854 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc85c10a3d0 con 0x7fc85c0ffd30 2023-07-14T17:42:56.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.360+0000 7fc8537fe700 1 -- 172.21.15.88:0/3107938854 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v86) v1 ==== 81+0+591 (secure 0 0 0) 0x7fc84c015260 con 0x7fc85c0ffd30 2023-07-14T17:42:56.342 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/1441439704 2023-07-15T17:36:43.799741+0000 2023-07-14T17:42:56.342 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/3897985004 2023-07-15T17:36:43.799741+0000 2023-07-14T17:42:56.343 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6801/2424184187 2023-07-15T17:36:43.799741+0000 2023-07-14T17:42:56.343 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6801/1369181448 2023-07-15T17:36:21.153428+0000 2023-07-14T17:42:56.343 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6800/1369181448 2023-07-15T17:36:21.153428+0000 2023-07-14T17:42:56.343 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/1790623516 2023-07-15T17:36:21.153428+0000 2023-07-14T17:42:56.343 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/3232411191 2023-07-15T17:36:21.153428+0000 2023-07-14T17:42:56.343 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/3329866608 2023-07-15T17:36:21.153428+0000 2023-07-14T17:42:56.343 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/954461155 2023-07-15T17:36:43.799741+0000 2023-07-14T17:42:56.343 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6800/2424184187 2023-07-15T17:36:43.799741+0000 2023-07-14T17:42:56.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.362+0000 7fc8646ad700 1 -- 172.21.15.88:0/3107938854 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc8480a1150 msgr2=0x7fc8480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:56.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.362+0000 7fc8646ad700 1 --2- 172.21.15.88:0/3107938854 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc8480a1150 0x7fc8480a3600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc85800c010 tx=0x7fc85800b540 comp rx=0 tx=0).stop 2023-07-14T17:42:56.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.362+0000 7fc8646ad700 1 -- 172.21.15.88:0/3107938854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc85c0ffd30 msgr2=0x7fc85c19a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:56.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.362+0000 7fc8646ad700 1 --2- 172.21.15.88:0/3107938854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc85c0ffd30 0x7fc85c19a5c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fc84c00b530 tx=0x7fc84c00b8f0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.362+0000 7fc8646ad700 1 -- 172.21.15.88:0/3107938854 shutdown_connections 2023-07-14T17:42:56.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.362+0000 7fc8646ad700 1 --2- 172.21.15.88:0/3107938854 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc85c1006c0 0x7fc85c19cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.362+0000 7fc8646ad700 1 --2- 172.21.15.88:0/3107938854 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc85c0feb40 0x7fc85c069340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.345 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.362+0000 7fc8646ad700 1 --2- 172.21.15.88:0/3107938854 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc8480a1150 0x7fc8480a3600 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.362+0000 7fc8646ad700 1 --2- 172.21.15.88:0/3107938854 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc85c0ffd30 0x7fc85c19a5c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.362+0000 7fc8646ad700 1 -- 172.21.15.88:0/3107938854 >> 172.21.15.88:0/3107938854 conn(0x7fc85c0fa160 msgr2=0x7fc85c0fc3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:56.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.362+0000 7fc8646ad700 1 -- 172.21.15.88:0/3107938854 shutdown_connections 2023-07-14T17:42:56.346 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.362+0000 7fc8646ad700 1 -- 172.21.15.88:0/3107938854 wait complete. 2023-07-14T17:42:56.346 INFO:teuthology.orchestra.run.smithi088.stderr:listed 10 entries 2023-07-14T17:42:56.356 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:42:56.356 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T17:42:56.373 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:42:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.552+0000 7f3746f54700 1 -- 172.21.15.88:0/2063931682 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37401017a0 msgr2=0x7f3740108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.552+0000 7f3746f54700 1 --2- 172.21.15.88:0/2063931682 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37401017a0 0x7f3740108060 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f373800b0a0 tx=0x7f373800b3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.552+0000 7f3746f54700 1 -- 172.21.15.88:0/2063931682 shutdown_connections 2023-07-14T17:42:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.552+0000 7f3746f54700 1 --2- 172.21.15.88:0/2063931682 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f37401017a0 0x7f3740108060 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.552+0000 7f3746f54700 1 --2- 172.21.15.88:0/2063931682 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37400ffc20 0x7f3740100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.552+0000 7f3746f54700 1 --2- 172.21.15.88:0/2063931682 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3740100e10 0x7f3740101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.552+0000 7f3746f54700 1 -- 172.21.15.88:0/2063931682 >> 172.21.15.88:0/2063931682 conn(0x7f37400fb260 msgr2=0x7f37400fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:56.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.552+0000 7f3746f54700 1 -- 172.21.15.88:0/2063931682 shutdown_connections 2023-07-14T17:42:56.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.552+0000 7f3746f54700 1 -- 172.21.15.88:0/2063931682 wait complete. 2023-07-14T17:42:56.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f3746f54700 1 Processor -- start 2023-07-14T17:42:56.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f3746f54700 1 -- start start 2023-07-14T17:42:56.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f3746f54700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37400ffc20 0x7f37401958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f3746f54700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3740100e10 0x7f3740195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f3746f54700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37401017a0 0x7f374019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f3746f54700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374019d470 con 0x7f3740100e10 2023-07-14T17:42:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f3746f54700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374019d5b0 con 0x7f37401017a0 2023-07-14T17:42:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f3746f54700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374019d740 con 0x7f37400ffc20 2023-07-14T17:42:56.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f373ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3740100e10 0x7f3740195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:56.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f373ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3740100e10 0x7f3740195e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41724/0 (socket says 172.21.15.88:41724) 2023-07-14T17:42:56.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f373ffff700 1 -- 172.21.15.88:0/594658864 learned_addr learned my addr 172.21.15.88:0/594658864 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:56.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f3744cf0700 1 --2- 172.21.15.88:0/594658864 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37400ffc20 0x7f37401958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:56.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f37454f1700 1 --2- 172.21.15.88:0/594658864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37401017a0 0x7f374019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:56.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f373ffff700 1 -- 172.21.15.88:0/594658864 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37400ffc20 msgr2=0x7f37401958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:56.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f373ffff700 1 --2- 172.21.15.88:0/594658864 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37400ffc20 0x7f37401958f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f373ffff700 1 -- 172.21.15.88:0/594658864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37401017a0 msgr2=0x7f374019cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:56.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f373ffff700 1 --2- 172.21.15.88:0/594658864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37401017a0 0x7f374019cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f373ffff700 1 -- 172.21.15.88:0/594658864 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3738009d00 con 0x7f3740100e10 2023-07-14T17:42:56.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.553+0000 7f373ffff700 1 --2- 172.21.15.88:0/594658864 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3740100e10 0x7f3740195e30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f372c00d950 tx=0x7f372c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:56.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.554+0000 7f373dffb700 1 -- 172.21.15.88:0/594658864 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372c009940 con 0x7f3740100e10 2023-07-14T17:42:56.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.554+0000 7f373dffb700 1 -- 172.21.15.88:0/594658864 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f372c010460 con 0x7f3740100e10 2023-07-14T17:42:56.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.554+0000 7f3746f54700 1 -- 172.21.15.88:0/594658864 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f374019da20 con 0x7f3740100e10 2023-07-14T17:42:56.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.554+0000 7f373dffb700 1 -- 172.21.15.88:0/594658864 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372c00f620 con 0x7f3740100e10 2023-07-14T17:42:56.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.554+0000 7f3746f54700 1 -- 172.21.15.88:0/594658864 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f374019df80 con 0x7f3740100e10 2023-07-14T17:42:56.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.555+0000 7f3746f54700 1 -- 172.21.15.88:0/594658864 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3740107140 con 0x7f3740100e10 2023-07-14T17:42:56.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.556+0000 7f373dffb700 1 -- 172.21.15.88:0/594658864 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f372c0105d0 con 0x7f3740100e10 2023-07-14T17:42:56.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.556+0000 7f373dffb700 1 --2- 172.21.15.88:0/594658864 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f37300a1220 0x7f37300a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:56.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.556+0000 7f373dffb700 1 -- 172.21.15.88:0/594658864 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f372c0cc8d0 con 0x7f3740100e10 2023-07-14T17:42:56.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.556+0000 7f3744cf0700 1 --2- 172.21.15.88:0/594658864 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f37300a1220 0x7f37300a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:56.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.557+0000 7f3744cf0700 1 --2- 172.21.15.88:0/594658864 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f37300a1220 0x7f37300a36d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3740100c70 tx=0x7f373400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:56.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.560+0000 7f373dffb700 1 -- 172.21.15.88:0/594658864 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f372c095e30 con 0x7f3740100e10 2023-07-14T17:42:56.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.753+0000 7f3746f54700 1 -- 172.21.15.88:0/594658864 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f374002c8e0 con 0x7f3740100e10 2023-07-14T17:42:56.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.755+0000 7f373dffb700 1 -- 172.21.15.88:0/594658864 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23985 (secure 0 0 0) 0x7f372c016090 con 0x7f3740100e10 2023-07-14T17:42:56.737 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:56.737 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":86,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:42:46.934364+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:42:56.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.757+0000 7f3746f54700 1 -- 172.21.15.88:0/594658864 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f37300a1220 msgr2=0x7f37300a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:56.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.757+0000 7f3746f54700 1 --2- 172.21.15.88:0/594658864 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f37300a1220 0x7f37300a36d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3740100c70 tx=0x7f373400b410 comp rx=0 tx=0).stop 2023-07-14T17:42:56.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.757+0000 7f3746f54700 1 -- 172.21.15.88:0/594658864 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3740100e10 msgr2=0x7f3740195e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:56.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.757+0000 7f3746f54700 1 --2- 172.21.15.88:0/594658864 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3740100e10 0x7f3740195e30 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f372c00d950 tx=0x7f372c00dc60 comp rx=0 tx=0).stop 2023-07-14T17:42:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.757+0000 7f3746f54700 1 -- 172.21.15.88:0/594658864 shutdown_connections 2023-07-14T17:42:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.757+0000 7f3746f54700 1 --2- 172.21.15.88:0/594658864 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f37401017a0 0x7f374019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.757+0000 7f3746f54700 1 --2- 172.21.15.88:0/594658864 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f37400ffc20 0x7f37401958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.757+0000 7f3746f54700 1 --2- 172.21.15.88:0/594658864 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f37300a1220 0x7f37300a36d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.757+0000 7f3746f54700 1 --2- 172.21.15.88:0/594658864 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3740100e10 0x7f3740195e30 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.757+0000 7f3746f54700 1 -- 172.21.15.88:0/594658864 >> 172.21.15.88:0/594658864 conn(0x7f37400fb260 msgr2=0x7f3740104280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:56.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.757+0000 7f3746f54700 1 -- 172.21.15.88:0/594658864 shutdown_connections 2023-07-14T17:42:56.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.757+0000 7f3746f54700 1 -- 172.21.15.88:0/594658864 wait complete. 2023-07-14T17:42:56.751 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T17:38:50.126535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T17:42:40.725870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T17:42:41.512749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T17:42:44.764573+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T17:42:56.752 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T17:42:56.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.893+0000 7ff818a94700 1 -- 172.21.15.88:0/1091485570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8141001c0 msgr2=0x7ff8141005d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:56.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.893+0000 7ff818a94700 1 --2- 172.21.15.88:0/1091485570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8141001c0 0x7ff8141005d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff808009990 tx=0x7ff808009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.893+0000 7ff818a94700 1 -- 172.21.15.88:0/1091485570 shutdown_connections 2023-07-14T17:42:56.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.893+0000 7ff818a94700 1 --2- 172.21.15.88:0/1091485570 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8140ff730 0x7ff814106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.893+0000 7ff818a94700 1 --2- 172.21.15.88:0/1091485570 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8141001c0 0x7ff8141005d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.893+0000 7ff818a94700 1 --2- 172.21.15.88:0/1091485570 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8140fed80 0x7ff8140ff1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.893+0000 7ff818a94700 1 -- 172.21.15.88:0/1091485570 >> 172.21.15.88:0/1091485570 conn(0x7ff8140f98e0 msgr2=0x7ff8140fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:56.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.893+0000 7ff818a94700 1 -- 172.21.15.88:0/1091485570 shutdown_connections 2023-07-14T17:42:56.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.893+0000 7ff818a94700 1 -- 172.21.15.88:0/1091485570 wait complete. 2023-07-14T17:42:56.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.894+0000 7ff818a94700 1 Processor -- start 2023-07-14T17:42:56.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.894+0000 7ff818a94700 1 -- start start 2023-07-14T17:42:56.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.894+0000 7ff818a94700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8140fed80 0x7ff8141958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:56.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.894+0000 7ff818a94700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8140ff730 0x7ff814195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:56.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.894+0000 7ff818a94700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8141001c0 0x7ff81419ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:56.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.894+0000 7ff818a94700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81419d430 con 0x7ff8141001c0 2023-07-14T17:42:56.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.894+0000 7ff818a94700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81419d570 con 0x7ff8140fed80 2023-07-14T17:42:56.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.894+0000 7ff818a94700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff81419d700 con 0x7ff8140ff730 2023-07-14T17:42:56.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.894+0000 7ff812d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8141001c0 0x7ff81419ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:56.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.894+0000 7ff812d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8141001c0 0x7ff81419ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41740/0 (socket says 172.21.15.88:41740) 2023-07-14T17:42:56.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.894+0000 7ff81259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8140fed80 0x7ff8141958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:56.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.894+0000 7ff812d9d700 1 -- 172.21.15.88:0/1304689573 learned_addr learned my addr 172.21.15.88:0/1304689573 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:56.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.895+0000 7ff811d9b700 1 --2- 172.21.15.88:0/1304689573 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8140ff730 0x7ff814195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:56.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.895+0000 7ff812d9d700 1 -- 172.21.15.88:0/1304689573 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8140ff730 msgr2=0x7ff814195df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:56.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.895+0000 7ff812d9d700 1 --2- 172.21.15.88:0/1304689573 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8140ff730 0x7ff814195df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.895+0000 7ff812d9d700 1 -- 172.21.15.88:0/1304689573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8140fed80 msgr2=0x7ff8141958b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:56.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.895+0000 7ff812d9d700 1 --2- 172.21.15.88:0/1304689573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8140fed80 0x7ff8141958b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:56.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.895+0000 7ff812d9d700 1 -- 172.21.15.88:0/1304689573 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8080095f0 con 0x7ff8141001c0 2023-07-14T17:42:56.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.895+0000 7ff811d9b700 1 --2- 172.21.15.88:0/1304689573 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8140ff730 0x7ff814195df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:42:56.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.895+0000 7ff812d9d700 1 --2- 172.21.15.88:0/1304689573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8141001c0 0x7ff81419ce60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff80c007ee0 tx=0x7ff80c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:56.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.895+0000 7ff8037fe700 1 -- 172.21.15.88:0/1304689573 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80c017070 con 0x7ff8141001c0 2023-07-14T17:42:56.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.895+0000 7ff818a94700 1 -- 172.21.15.88:0/1304689573 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff81419d9e0 con 0x7ff8141001c0 2023-07-14T17:42:56.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.895+0000 7ff8037fe700 1 -- 172.21.15.88:0/1304689573 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff80c00f040 con 0x7ff8141001c0 2023-07-14T17:42:56.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.895+0000 7ff8037fe700 1 -- 172.21.15.88:0/1304689573 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff80c005710 con 0x7ff8141001c0 2023-07-14T17:42:56.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.895+0000 7ff818a94700 1 -- 172.21.15.88:0/1304689573 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff81419df70 con 0x7ff8141001c0 2023-07-14T17:42:56.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.896+0000 7ff818a94700 1 -- 172.21.15.88:0/1304689573 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff81410a240 con 0x7ff8141001c0 2023-07-14T17:42:56.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.897+0000 7ff8037fe700 1 -- 172.21.15.88:0/1304689573 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff80c005870 con 0x7ff8141001c0 2023-07-14T17:42:56.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.897+0000 7ff8037fe700 1 --2- 172.21.15.88:0/1304689573 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff7fc0a1150 0x7ff7fc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:56.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.898+0000 7ff81259c700 1 --2- 172.21.15.88:0/1304689573 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff7fc0a1150 0x7ff7fc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:56.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.898+0000 7ff8037fe700 1 -- 172.21.15.88:0/1304689573 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff80c0cc370 con 0x7ff8141001c0 2023-07-14T17:42:56.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.898+0000 7ff81259c700 1 --2- 172.21.15.88:0/1304689573 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff7fc0a1150 0x7ff7fc0a3600 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff80800b5c0 tx=0x7ff808005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:56.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:56.902+0000 7ff8037fe700 1 -- 172.21.15.88:0/1304689573 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff80c095870 con 0x7ff8141001c0 2023-07-14T17:42:57.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.094+0000 7ff818a94700 1 -- 172.21.15.88:0/1304689573 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff81419e580 con 0x7ff8141001c0 2023-07-14T17:42:57.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.094+0000 7ff8037fe700 1 -- 172.21.15.88:0/1304689573 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff80c098e90 con 0x7ff8141001c0 2023-07-14T17:42:57.077 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-07-14T17:42:57.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.096+0000 7ff818a94700 1 -- 172.21.15.88:0/1304689573 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff7fc0a1150 msgr2=0x7ff7fc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:57.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.096+0000 7ff818a94700 1 --2- 172.21.15.88:0/1304689573 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff7fc0a1150 0x7ff7fc0a3600 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff80800b5c0 tx=0x7ff808005fb0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.097+0000 7ff818a94700 1 -- 172.21.15.88:0/1304689573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8141001c0 msgr2=0x7ff81419ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:57.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.097+0000 7ff818a94700 1 --2- 172.21.15.88:0/1304689573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8141001c0 0x7ff81419ce60 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff80c007ee0 tx=0x7ff80c00d3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.097+0000 7ff818a94700 1 -- 172.21.15.88:0/1304689573 shutdown_connections 2023-07-14T17:42:57.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.097+0000 7ff818a94700 1 --2- 172.21.15.88:0/1304689573 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff7fc0a1150 0x7ff7fc0a3600 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.097+0000 7ff818a94700 1 --2- 172.21.15.88:0/1304689573 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8141001c0 0x7ff81419ce60 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.097+0000 7ff818a94700 1 --2- 172.21.15.88:0/1304689573 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8140fed80 0x7ff8141958b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.097+0000 7ff818a94700 1 --2- 172.21.15.88:0/1304689573 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8140ff730 0x7ff814195df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.097+0000 7ff818a94700 1 -- 172.21.15.88:0/1304689573 >> 172.21.15.88:0/1304689573 conn(0x7ff8140f98e0 msgr2=0x7ff8140fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:57.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.097+0000 7ff818a94700 1 -- 172.21.15.88:0/1304689573 shutdown_connections 2023-07-14T17:42:57.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.097+0000 7ff818a94700 1 -- 172.21.15.88:0/1304689573 wait complete. 2023-07-14T17:42:57.090 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:56 smithi088 ceph-mon[127465]: pgmap v262: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 328 B/s wr, 0 op/s 2023-07-14T17:42:57.090 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:56 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1646255181' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:42:57.091 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:56 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3107938854' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T17:42:57.091 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:56 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/594658864' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:57.092 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T17:42:57.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:56 smithi178 ceph-mon[129368]: pgmap v262: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 328 B/s wr, 0 op/s 2023-07-14T17:42:57.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:56 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1646255181' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:42:57.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:56 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3107938854' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T17:42:57.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:56 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/594658864' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:57.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.235+0000 7ff99aa46700 1 -- 172.21.15.88:0/948523904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff994105930 msgr2=0x7ff994107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:57.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.235+0000 7ff99aa46700 1 --2- 172.21.15.88:0/948523904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff994105930 0x7ff994107d30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff98c00b0a0 tx=0x7ff98c00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.235+0000 7ff99aa46700 1 -- 172.21.15.88:0/948523904 shutdown_connections 2023-07-14T17:42:57.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.235+0000 7ff99aa46700 1 --2- 172.21.15.88:0/948523904 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff994105930 0x7ff994107d30 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.235+0000 7ff99aa46700 1 --2- 172.21.15.88:0/948523904 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff9941006f0 0x7ff994102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.235+0000 7ff99aa46700 1 --2- 172.21.15.88:0/948523904 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff994103010 0x7ff9941053f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.235+0000 7ff99aa46700 1 -- 172.21.15.88:0/948523904 >> 172.21.15.88:0/948523904 conn(0x7ff9940fa130 msgr2=0x7ff9940fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:57.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.235+0000 7ff99aa46700 1 -- 172.21.15.88:0/948523904 shutdown_connections 2023-07-14T17:42:57.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.235+0000 7ff99aa46700 1 -- 172.21.15.88:0/948523904 wait complete. 2023-07-14T17:42:57.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff99aa46700 1 Processor -- start 2023-07-14T17:42:57.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff99aa46700 1 -- start start 2023-07-14T17:42:57.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff99aa46700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff9941006f0 0x7ff9941959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:57.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff99aa46700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff994103010 0x7ff994195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:57.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff99aa46700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff994105930 0x7ff99419cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:57.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff99aa46700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99419d540 con 0x7ff994103010 2023-07-14T17:42:57.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff99aa46700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99419d680 con 0x7ff994105930 2023-07-14T17:42:57.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff99aa46700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff99419d810 con 0x7ff9941006f0 2023-07-14T17:42:57.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff98bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff994103010 0x7ff994195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:57.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff98bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff994103010 0x7ff994195f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41752/0 (socket says 172.21.15.88:41752) 2023-07-14T17:42:57.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff98bfff700 1 -- 172.21.15.88:0/1674934648 learned_addr learned my addr 172.21.15.88:0/1674934648 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:57.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff993fff700 1 --2- 172.21.15.88:0/1674934648 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff9941006f0 0x7ff9941959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:57.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff998fe3700 1 --2- 172.21.15.88:0/1674934648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff994105930 0x7ff99419cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:57.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff98bfff700 1 -- 172.21.15.88:0/1674934648 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff9941006f0 msgr2=0x7ff9941959c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:57.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff98bfff700 1 --2- 172.21.15.88:0/1674934648 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff9941006f0 0x7ff9941959c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff98bfff700 1 -- 172.21.15.88:0/1674934648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff994105930 msgr2=0x7ff99419cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:57.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff98bfff700 1 --2- 172.21.15.88:0/1674934648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff994105930 0x7ff99419cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff98bfff700 1 -- 172.21.15.88:0/1674934648 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98c009d00 con 0x7ff994103010 2023-07-14T17:42:57.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff998fe3700 1 --2- 172.21.15.88:0/1674934648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff994105930 0x7ff99419cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:42:57.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.236+0000 7ff98bfff700 1 --2- 172.21.15.88:0/1674934648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff994103010 0x7ff994195f00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff97c00c920 tx=0x7ff97c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:57.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.237+0000 7ff991ffb700 1 -- 172.21.15.88:0/1674934648 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97c007b00 con 0x7ff994103010 2023-07-14T17:42:57.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.237+0000 7ff99aa46700 1 -- 172.21.15.88:0/1674934648 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff99419daf0 con 0x7ff994103010 2023-07-14T17:42:57.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.237+0000 7ff991ffb700 1 -- 172.21.15.88:0/1674934648 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff97c00ce70 con 0x7ff994103010 2023-07-14T17:42:57.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.237+0000 7ff991ffb700 1 -- 172.21.15.88:0/1674934648 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97c0053e0 con 0x7ff994103010 2023-07-14T17:42:57.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.237+0000 7ff99aa46700 1 -- 172.21.15.88:0/1674934648 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff99419e000 con 0x7ff994103010 2023-07-14T17:42:57.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.238+0000 7ff99aa46700 1 -- 172.21.15.88:0/1674934648 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9940fbca0 con 0x7ff994103010 2023-07-14T17:42:57.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.238+0000 7ff991ffb700 1 -- 172.21.15.88:0/1674934648 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff97c005580 con 0x7ff994103010 2023-07-14T17:42:57.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.239+0000 7ff991ffb700 1 --2- 172.21.15.88:0/1674934648 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff9800a1150 0x7ff9800a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:57.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:56 smithi107 ceph-mon[128999]: pgmap v262: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 328 B/s wr, 0 op/s 2023-07-14T17:42:57.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:56 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1646255181' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:42:57.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:56 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3107938854' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T17:42:57.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:56 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/594658864' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:42:57.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.239+0000 7ff991ffb700 1 -- 172.21.15.88:0/1674934648 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff97c0cc6f0 con 0x7ff994103010 2023-07-14T17:42:57.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.239+0000 7ff993fff700 1 --2- 172.21.15.88:0/1674934648 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff9800a1150 0x7ff9800a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:57.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.239+0000 7ff993fff700 1 --2- 172.21.15.88:0/1674934648 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff9800a1150 0x7ff9800a3600 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff984009a10 tx=0x7ff984009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:57.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.243+0000 7ff991ffb700 1 -- 172.21.15.88:0/1674934648 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff97c095c50 con 0x7ff994103010 2023-07-14T17:42:57.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.431+0000 7ff99aa46700 1 -- 172.21.15.88:0/1674934648 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff9940612f0 con 0x7ff994103010 2023-07-14T17:42:57.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.432+0000 7ff991ffb700 1 -- 172.21.15.88:0/1674934648 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v86) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff97c099270 con 0x7ff994103010 2023-07-14T17:42:57.415 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-07-14T17:42:57.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.434+0000 7ff99aa46700 1 -- 172.21.15.88:0/1674934648 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff9800a1150 msgr2=0x7ff9800a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:57.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.434+0000 7ff99aa46700 1 --2- 172.21.15.88:0/1674934648 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff9800a1150 0x7ff9800a3600 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff984009a10 tx=0x7ff984009190 comp rx=0 tx=0).stop 2023-07-14T17:42:57.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.434+0000 7ff99aa46700 1 -- 172.21.15.88:0/1674934648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff994103010 msgr2=0x7ff994195f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:57.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.434+0000 7ff99aa46700 1 --2- 172.21.15.88:0/1674934648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff994103010 0x7ff994195f00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff97c00c920 tx=0x7ff97c00cc30 comp rx=0 tx=0).stop 2023-07-14T17:42:57.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.434+0000 7ff99aa46700 1 -- 172.21.15.88:0/1674934648 shutdown_connections 2023-07-14T17:42:57.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.434+0000 7ff99aa46700 1 --2- 172.21.15.88:0/1674934648 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff994105930 0x7ff99419cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.434+0000 7ff99aa46700 1 --2- 172.21.15.88:0/1674934648 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff9941006f0 0x7ff9941959c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.434+0000 7ff99aa46700 1 --2- 172.21.15.88:0/1674934648 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff9800a1150 0x7ff9800a3600 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.434+0000 7ff99aa46700 1 --2- 172.21.15.88:0/1674934648 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff994103010 0x7ff994195f00 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.434+0000 7ff99aa46700 1 -- 172.21.15.88:0/1674934648 >> 172.21.15.88:0/1674934648 conn(0x7ff9940fa130 msgr2=0x7ff994101350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:57.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.434+0000 7ff99aa46700 1 -- 172.21.15.88:0/1674934648 shutdown_connections 2023-07-14T17:42:57.418 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.434+0000 7ff99aa46700 1 -- 172.21.15.88:0/1674934648 wait complete. 2023-07-14T17:42:57.428 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T17:42:57.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.570+0000 7fd26d531700 1 -- 172.21.15.88:0/2975565354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd268100f90 msgr2=0x7fd2681013e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:57.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.570+0000 7fd26d531700 1 --2- 172.21.15.88:0/2975565354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd268100f90 0x7fd2681013e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd25c009990 tx=0x7fd25c009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.570+0000 7fd26d531700 1 -- 172.21.15.88:0/2975565354 shutdown_connections 2023-07-14T17:42:57.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.570+0000 7fd26d531700 1 --2- 172.21.15.88:0/2975565354 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd268101920 0x7fd2681081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.570+0000 7fd26d531700 1 --2- 172.21.15.88:0/2975565354 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd2680fdf20 0x7fd2680fe330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.570+0000 7fd26d531700 1 --2- 172.21.15.88:0/2975565354 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd268100f90 0x7fd2681013e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.570+0000 7fd26d531700 1 -- 172.21.15.88:0/2975565354 >> 172.21.15.88:0/2975565354 conn(0x7fd2680f98e0 msgr2=0x7fd2680fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:57.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.570+0000 7fd26d531700 1 -- 172.21.15.88:0/2975565354 shutdown_connections 2023-07-14T17:42:57.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.570+0000 7fd26d531700 1 -- 172.21.15.88:0/2975565354 wait complete. 2023-07-14T17:42:57.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.571+0000 7fd26d531700 1 Processor -- start 2023-07-14T17:42:57.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.571+0000 7fd26d531700 1 -- start start 2023-07-14T17:42:57.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.571+0000 7fd26d531700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2680fdf20 0x7fd2681958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:57.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.571+0000 7fd26d531700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd268100f90 0x7fd268195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:57.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.571+0000 7fd26d531700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd268101920 0x7fd26819ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:57.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.571+0000 7fd26d531700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26819d420 con 0x7fd2680fdf20 2023-07-14T17:42:57.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.571+0000 7fd26d531700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26819d560 con 0x7fd268100f90 2023-07-14T17:42:57.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.571+0000 7fd26d531700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26819d6f0 con 0x7fd268101920 2023-07-14T17:42:57.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.571+0000 7fd266ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2680fdf20 0x7fd2681958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:57.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.571+0000 7fd266ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2680fdf20 0x7fd2681958a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41764/0 (socket says 172.21.15.88:41764) 2023-07-14T17:42:57.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.572+0000 7fd266ffd700 1 -- 172.21.15.88:0/1524476230 learned_addr learned my addr 172.21.15.88:0/1524476230 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:57.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.572+0000 7fd2667fc700 1 --2- 172.21.15.88:0/1524476230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd268100f90 0x7fd268195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:57.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.572+0000 7fd2677fe700 1 --2- 172.21.15.88:0/1524476230 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd268101920 0x7fd26819ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:57.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.572+0000 7fd266ffd700 1 -- 172.21.15.88:0/1524476230 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd268101920 msgr2=0x7fd26819ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:57.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.572+0000 7fd266ffd700 1 --2- 172.21.15.88:0/1524476230 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd268101920 0x7fd26819ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.572+0000 7fd266ffd700 1 -- 172.21.15.88:0/1524476230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd268100f90 msgr2=0x7fd268195de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:57.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.572+0000 7fd266ffd700 1 --2- 172.21.15.88:0/1524476230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd268100f90 0x7fd268195de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.572+0000 7fd266ffd700 1 -- 172.21.15.88:0/1524476230 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25c0095f0 con 0x7fd2680fdf20 2023-07-14T17:42:57.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.572+0000 7fd2667fc700 1 --2- 172.21.15.88:0/1524476230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd268100f90 0x7fd268195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:42:57.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.572+0000 7fd266ffd700 1 --2- 172.21.15.88:0/1524476230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2680fdf20 0x7fd2681958a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd25800d980 tx=0x7fd25800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:57.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.572+0000 7fd24ffff700 1 -- 172.21.15.88:0/1524476230 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25800de60 con 0x7fd2680fdf20 2023-07-14T17:42:57.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.572+0000 7fd26d531700 1 -- 172.21.15.88:0/1524476230 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd26819d9d0 con 0x7fd2680fdf20 2023-07-14T17:42:57.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.572+0000 7fd24ffff700 1 -- 172.21.15.88:0/1524476230 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd2580044b0 con 0x7fd2680fdf20 2023-07-14T17:42:57.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.572+0000 7fd24ffff700 1 -- 172.21.15.88:0/1524476230 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd258003b40 con 0x7fd2680fdf20 2023-07-14T17:42:57.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.572+0000 7fd26d531700 1 -- 172.21.15.88:0/1524476230 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd26819df30 con 0x7fd2680fdf20 2023-07-14T17:42:57.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.573+0000 7fd26d531700 1 -- 172.21.15.88:0/1524476230 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd268106d50 con 0x7fd2680fdf20 2023-07-14T17:42:57.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.574+0000 7fd24ffff700 1 -- 172.21.15.88:0/1524476230 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd258004870 con 0x7fd2680fdf20 2023-07-14T17:42:57.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.575+0000 7fd24ffff700 1 --2- 172.21.15.88:0/1524476230 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd2500a1150 0x7fd2500a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:57.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.575+0000 7fd2667fc700 1 --2- 172.21.15.88:0/1524476230 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd2500a1150 0x7fd2500a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:57.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.575+0000 7fd24ffff700 1 -- 172.21.15.88:0/1524476230 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd2580ccc10 con 0x7fd2680fdf20 2023-07-14T17:42:57.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.575+0000 7fd2667fc700 1 --2- 172.21.15.88:0/1524476230 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd2500a1150 0x7fd2500a3600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd25c00b5c0 tx=0x7fd25c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:57.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.578+0000 7fd24ffff700 1 -- 172.21.15.88:0/1524476230 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd258096110 con 0x7fd2680fdf20 2023-07-14T17:42:57.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.765+0000 7fd26d531700 1 -- 172.21.15.88:0/1524476230 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd26802cb50 con 0x7fd2680fdf20 2023-07-14T17:42:57.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.765+0000 7fd24ffff700 1 -- 172.21.15.88:0/1524476230 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v86) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd258009940 con 0x7fd2680fdf20 2023-07-14T17:42:57.748 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 64 2023-07-14T17:42:57.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.767+0000 7fd26d531700 1 -- 172.21.15.88:0/1524476230 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd2500a1150 msgr2=0x7fd2500a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:57.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.767+0000 7fd26d531700 1 --2- 172.21.15.88:0/1524476230 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd2500a1150 0x7fd2500a3600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd25c00b5c0 tx=0x7fd25c005fb0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.767+0000 7fd26d531700 1 -- 172.21.15.88:0/1524476230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2680fdf20 msgr2=0x7fd2681958a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:57.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.767+0000 7fd26d531700 1 --2- 172.21.15.88:0/1524476230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2680fdf20 0x7fd2681958a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd25800d980 tx=0x7fd25800dd40 comp rx=0 tx=0).stop 2023-07-14T17:42:57.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.767+0000 7fd26d531700 1 -- 172.21.15.88:0/1524476230 shutdown_connections 2023-07-14T17:42:57.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.767+0000 7fd26d531700 1 --2- 172.21.15.88:0/1524476230 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd268101920 0x7fd26819ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.767+0000 7fd26d531700 1 --2- 172.21.15.88:0/1524476230 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd2500a1150 0x7fd2500a3600 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.767+0000 7fd26d531700 1 --2- 172.21.15.88:0/1524476230 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd2680fdf20 0x7fd2681958a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.767+0000 7fd26d531700 1 --2- 172.21.15.88:0/1524476230 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd268100f90 0x7fd268195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.767+0000 7fd26d531700 1 -- 172.21.15.88:0/1524476230 >> 172.21.15.88:0/1524476230 conn(0x7fd2680f98e0 msgr2=0x7fd2680fbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:57.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.768+0000 7fd26d531700 1 -- 172.21.15.88:0/1524476230 shutdown_connections 2023-07-14T17:42:57.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.768+0000 7fd26d531700 1 -- 172.21.15.88:0/1524476230 wait complete. 2023-07-14T17:42:57.762 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T17:42:57.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.904+0000 7f178c207700 1 -- 172.21.15.88:0/3601099536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17840ff470 msgr2=0x7f17840ff8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.904+0000 7f178c207700 1 --2- 172.21.15.88:0/3601099536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17840ff470 0x7f17840ff8c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f1778009990 tx=0x7f1778009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.904+0000 7f178c207700 1 -- 172.21.15.88:0/3601099536 shutdown_connections 2023-07-14T17:42:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.904+0000 7f178c207700 1 --2- 172.21.15.88:0/3601099536 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17840ffe00 0x7f17841066c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.904+0000 7f178c207700 1 --2- 172.21.15.88:0/3601099536 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17840fe280 0x7f17840fe690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.904+0000 7f178c207700 1 --2- 172.21.15.88:0/3601099536 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17840ff470 0x7f17840ff8c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.904+0000 7f178c207700 1 -- 172.21.15.88:0/3601099536 >> 172.21.15.88:0/3601099536 conn(0x7f17840f98a0 msgr2=0x7f17840fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:57.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.904+0000 7f178c207700 1 -- 172.21.15.88:0/3601099536 shutdown_connections 2023-07-14T17:42:57.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.904+0000 7f178c207700 1 -- 172.21.15.88:0/3601099536 wait complete. 2023-07-14T17:42:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.905+0000 7f178c207700 1 Processor -- start 2023-07-14T17:42:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.905+0000 7f178c207700 1 -- start start 2023-07-14T17:42:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.905+0000 7f178c207700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17840fe280 0x7f17841958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.905+0000 7f178c207700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17840ff470 0x7f1784195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.905+0000 7f178c207700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17840ffe00 0x7f178419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.905+0000 7f178c207700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178419d450 con 0x7f17840ffe00 2023-07-14T17:42:57.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.905+0000 7f178c207700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178419d590 con 0x7f17840fe280 2023-07-14T17:42:57.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.905+0000 7f178c207700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178419d720 con 0x7f17840ff470 2023-07-14T17:42:57.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.905+0000 7f178a7a4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17840ffe00 0x7f178419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:57.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.906+0000 7f178a7a4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17840ffe00 0x7f178419ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41782/0 (socket says 172.21.15.88:41782) 2023-07-14T17:42:57.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.906+0000 7f178a7a4700 1 -- 172.21.15.88:0/1764339983 learned_addr learned my addr 172.21.15.88:0/1764339983 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:57.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.906+0000 7f1789fa3700 1 --2- 172.21.15.88:0/1764339983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17840fe280 0x7f17841958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:57.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.906+0000 7f17897a2700 1 --2- 172.21.15.88:0/1764339983 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17840ff470 0x7f1784195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:57.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.906+0000 7f178a7a4700 1 -- 172.21.15.88:0/1764339983 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17840ff470 msgr2=0x7f1784195e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:57.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.906+0000 7f178a7a4700 1 --2- 172.21.15.88:0/1764339983 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17840ff470 0x7f1784195e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.906+0000 7f178a7a4700 1 -- 172.21.15.88:0/1764339983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17840fe280 msgr2=0x7f17841958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:57.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.906+0000 7f178a7a4700 1 --2- 172.21.15.88:0/1764339983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17840fe280 0x7f17841958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:57.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.906+0000 7f178a7a4700 1 -- 172.21.15.88:0/1764339983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17780095f0 con 0x7f17840ffe00 2023-07-14T17:42:57.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.906+0000 7f178a7a4700 1 --2- 172.21.15.88:0/1764339983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17840ffe00 0x7f178419ce80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f177c007ee0 tx=0x7f177c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:57.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.906+0000 7f1776ffd700 1 -- 172.21.15.88:0/1764339983 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c017070 con 0x7f17840ffe00 2023-07-14T17:42:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.906+0000 7f1776ffd700 1 -- 172.21.15.88:0/1764339983 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f177c00f040 con 0x7f17840ffe00 2023-07-14T17:42:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.906+0000 7f1776ffd700 1 -- 172.21.15.88:0/1764339983 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c005710 con 0x7f17840ffe00 2023-07-14T17:42:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.906+0000 7f178c207700 1 -- 172.21.15.88:0/1764339983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178419da00 con 0x7f17840ffe00 2023-07-14T17:42:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.906+0000 7f178c207700 1 -- 172.21.15.88:0/1764339983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f178419dec0 con 0x7f17840ffe00 2023-07-14T17:42:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.907+0000 7f178c207700 1 -- 172.21.15.88:0/1764339983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f178410b340 con 0x7f17840ffe00 2023-07-14T17:42:57.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.908+0000 7f1776ffd700 1 -- 172.21.15.88:0/1764339983 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f177c005870 con 0x7f17840ffe00 2023-07-14T17:42:57.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.909+0000 7f1776ffd700 1 --2- 172.21.15.88:0/1764339983 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f17700a11d0 0x7f17700a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:57.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.909+0000 7f1789fa3700 1 --2- 172.21.15.88:0/1764339983 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f17700a11d0 0x7f17700a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:57.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.909+0000 7f1776ffd700 1 -- 172.21.15.88:0/1764339983 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f177c0cc840 con 0x7f17840ffe00 2023-07-14T17:42:57.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.909+0000 7f1789fa3700 1 --2- 172.21.15.88:0/1764339983 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f17700a11d0 0x7f17700a3680 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f17840ff2d0 tx=0x7f1780007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:57.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:57.912+0000 7f1776ffd700 1 -- 172.21.15.88:0/1764339983 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f177c095da0 con 0x7f17840ffe00 2023-07-14T17:42:58.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.103+0000 7f178c207700 1 -- 172.21.15.88:0/1764339983 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1784105230 con 0x7f17840ffe00 2023-07-14T17:42:58.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.104+0000 7f1776ffd700 1 -- 172.21.15.88:0/1764339983 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v86) v1 ==== 103+0+11 (secure 0 0 0) 0x7f177c0993c0 con 0x7f17840ffe00 2023-07-14T17:42:58.086 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-07-14T17:42:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.106+0000 7f178c207700 1 -- 172.21.15.88:0/1764339983 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f17700a11d0 msgr2=0x7f17700a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.106+0000 7f178c207700 1 --2- 172.21.15.88:0/1764339983 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f17700a11d0 0x7f17700a3680 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f17840ff2d0 tx=0x7f1780007400 comp rx=0 tx=0).stop 2023-07-14T17:42:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.106+0000 7f178c207700 1 -- 172.21.15.88:0/1764339983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17840ffe00 msgr2=0x7f178419ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.106+0000 7f178c207700 1 --2- 172.21.15.88:0/1764339983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17840ffe00 0x7f178419ce80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f177c007ee0 tx=0x7f177c00d3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.106+0000 7f178c207700 1 -- 172.21.15.88:0/1764339983 shutdown_connections 2023-07-14T17:42:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.106+0000 7f178c207700 1 --2- 172.21.15.88:0/1764339983 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f17700a11d0 0x7f17700a3680 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.106+0000 7f178c207700 1 --2- 172.21.15.88:0/1764339983 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f17840ffe00 0x7f178419ce80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.106+0000 7f178c207700 1 --2- 172.21.15.88:0/1764339983 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f17840fe280 0x7f17841958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.106+0000 7f178c207700 1 --2- 172.21.15.88:0/1764339983 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f17840ff470 0x7f1784195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.106+0000 7f178c207700 1 -- 172.21.15.88:0/1764339983 >> 172.21.15.88:0/1764339983 conn(0x7f17840f98a0 msgr2=0x7f17840fbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.106+0000 7f178c207700 1 -- 172.21.15.88:0/1764339983 shutdown_connections 2023-07-14T17:42:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.106+0000 7f178c207700 1 -- 172.21.15.88:0/1764339983 wait complete. 2023-07-14T17:42:58.099 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:57 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1304689573' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:42:58.099 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:57 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1674934648' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:42:58.099 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:57 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1524476230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:42:58.100 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T17:42:58.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.239+0000 7efcb225f700 1 -- 172.21.15.88:0/925006683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcac1017c0 msgr2=0x7efcac108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:58.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.239+0000 7efcb225f700 1 --2- 172.21.15.88:0/925006683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcac1017c0 0x7efcac108080 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7efca400b0a0 tx=0x7efca400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.239+0000 7efcb225f700 1 -- 172.21.15.88:0/925006683 shutdown_connections 2023-07-14T17:42:58.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.239+0000 7efcb225f700 1 --2- 172.21.15.88:0/925006683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcac1017c0 0x7efcac108080 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.239+0000 7efcb225f700 1 --2- 172.21.15.88:0/925006683 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcac0ffc40 0x7efcac100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.239+0000 7efcb225f700 1 --2- 172.21.15.88:0/925006683 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcac100e30 0x7efcac101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.239+0000 7efcb225f700 1 -- 172.21.15.88:0/925006683 >> 172.21.15.88:0/925006683 conn(0x7efcac0fb240 msgr2=0x7efcac0fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:58.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.239+0000 7efcb225f700 1 -- 172.21.15.88:0/925006683 shutdown_connections 2023-07-14T17:42:58.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.239+0000 7efcb225f700 1 -- 172.21.15.88:0/925006683 wait complete. 2023-07-14T17:42:58.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.240+0000 7efcb225f700 1 Processor -- start 2023-07-14T17:42:58.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.240+0000 7efcb225f700 1 -- start start 2023-07-14T17:42:58.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.240+0000 7efcb225f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcac0ffc40 0x7efcac1959e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:58.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.240+0000 7efcb225f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcac100e30 0x7efcac195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:58.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.240+0000 7efcb225f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcac1017c0 0x7efcac19cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:58.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.240+0000 7efcb225f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac19d560 con 0x7efcac0ffc40 2023-07-14T17:42:58.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.240+0000 7efcb225f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac19d6a0 con 0x7efcac1017c0 2023-07-14T17:42:58.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.240+0000 7efcb225f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac19d830 con 0x7efcac100e30 2023-07-14T17:42:58.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.241+0000 7efcab7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcac0ffc40 0x7efcac1959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:58.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.241+0000 7efcab7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcac0ffc40 0x7efcac1959e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41798/0 (socket says 172.21.15.88:41798) 2023-07-14T17:42:58.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.241+0000 7efcab7fe700 1 -- 172.21.15.88:0/2592317775 learned_addr learned my addr 172.21.15.88:0/2592317775 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:58.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.241+0000 7efcabfff700 1 --2- 172.21.15.88:0/2592317775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcac1017c0 0x7efcac19cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:58.226 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.241+0000 7efcaaffd700 1 --2- 172.21.15.88:0/2592317775 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcac100e30 0x7efcac195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:58.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.241+0000 7efcab7fe700 1 -- 172.21.15.88:0/2592317775 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcac100e30 msgr2=0x7efcac195f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:58.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.241+0000 7efcab7fe700 1 --2- 172.21.15.88:0/2592317775 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcac100e30 0x7efcac195f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.241+0000 7efcab7fe700 1 -- 172.21.15.88:0/2592317775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcac1017c0 msgr2=0x7efcac19cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:58.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.241+0000 7efcab7fe700 1 --2- 172.21.15.88:0/2592317775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcac1017c0 0x7efcac19cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.241+0000 7efcab7fe700 1 -- 172.21.15.88:0/2592317775 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca4009d00 con 0x7efcac0ffc40 2023-07-14T17:42:58.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.241+0000 7efcaaffd700 1 --2- 172.21.15.88:0/2592317775 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcac100e30 0x7efcac195f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:42:58.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.241+0000 7efcabfff700 1 --2- 172.21.15.88:0/2592317775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcac1017c0 0x7efcac19cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:42:58.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.241+0000 7efcab7fe700 1 --2- 172.21.15.88:0/2592317775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcac0ffc40 0x7efcac1959e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00d980 tx=0x7efc9c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:58.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.241+0000 7efca8ff9700 1 -- 172.21.15.88:0/2592317775 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c00de60 con 0x7efcac0ffc40 2023-07-14T17:42:58.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.241+0000 7efcb225f700 1 -- 172.21.15.88:0/2592317775 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcac19db10 con 0x7efcac0ffc40 2023-07-14T17:42:58.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.241+0000 7efca8ff9700 1 -- 172.21.15.88:0/2592317775 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7efc9c0044b0 con 0x7efcac0ffc40 2023-07-14T17:42:58.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.241+0000 7efca8ff9700 1 -- 172.21.15.88:0/2592317775 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c003b40 con 0x7efcac0ffc40 2023-07-14T17:42:58.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.241+0000 7efcb225f700 1 -- 172.21.15.88:0/2592317775 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcac19e020 con 0x7efcac0ffc40 2023-07-14T17:42:58.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.242+0000 7efcb225f700 1 -- 172.21.15.88:0/2592317775 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcac107170 con 0x7efcac0ffc40 2023-07-14T17:42:58.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:57 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1304689573' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:42:58.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:57 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1674934648' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:42:58.229 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:57 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1524476230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:42:58.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.243+0000 7efca8ff9700 1 -- 172.21.15.88:0/2592317775 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7efc9c003ca0 con 0x7efcac0ffc40 2023-07-14T17:42:58.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.244+0000 7efca8ff9700 1 --2- 172.21.15.88:0/2592317775 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7efc940a1150 0x7efc940a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:58.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.244+0000 7efcaaffd700 1 --2- 172.21.15.88:0/2592317775 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7efc940a1150 0x7efc940a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:58.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.244+0000 7efca8ff9700 1 -- 172.21.15.88:0/2592317775 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7efc9c0cc890 con 0x7efcac0ffc40 2023-07-14T17:42:58.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.244+0000 7efcaaffd700 1 --2- 172.21.15.88:0/2592317775 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7efc940a1150 0x7efc940a3600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efca0009b50 tx=0x7efca0009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:58.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.248+0000 7efca8ff9700 1 -- 172.21.15.88:0/2592317775 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efc9c095d90 con 0x7efcac0ffc40 2023-07-14T17:42:58.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:57 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1304689573' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:42:58.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:57 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1674934648' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:42:58.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:57 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1524476230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:42:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.451+0000 7efcb225f700 1 -- 172.21.15.88:0/2592317775 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7efcac02c8e0 con 0x7efcac0ffc40 2023-07-14T17:42:58.433 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.451+0000 7efca8ff9700 1 -- 172.21.15.88:0/2592317775 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v86) v1 ==== 81+0+591 (secure 0 0 0) 0x7efc9c009940 con 0x7efcac0ffc40 2023-07-14T17:42:58.434 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/1441439704 2023-07-15T17:36:43.799741+0000 2023-07-14T17:42:58.434 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/3897985004 2023-07-15T17:36:43.799741+0000 2023-07-14T17:42:58.434 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6801/2424184187 2023-07-15T17:36:43.799741+0000 2023-07-14T17:42:58.434 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6801/1369181448 2023-07-15T17:36:21.153428+0000 2023-07-14T17:42:58.434 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6800/1369181448 2023-07-15T17:36:21.153428+0000 2023-07-14T17:42:58.434 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/1790623516 2023-07-15T17:36:21.153428+0000 2023-07-14T17:42:58.434 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/3232411191 2023-07-15T17:36:21.153428+0000 2023-07-14T17:42:58.435 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/3329866608 2023-07-15T17:36:21.153428+0000 2023-07-14T17:42:58.435 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/954461155 2023-07-15T17:36:43.799741+0000 2023-07-14T17:42:58.435 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6800/2424184187 2023-07-15T17:36:43.799741+0000 2023-07-14T17:42:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.453+0000 7efcb225f700 1 -- 172.21.15.88:0/2592317775 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7efc940a1150 msgr2=0x7efc940a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.453+0000 7efcb225f700 1 --2- 172.21.15.88:0/2592317775 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7efc940a1150 0x7efc940a3600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efca0009b50 tx=0x7efca0009550 comp rx=0 tx=0).stop 2023-07-14T17:42:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.453+0000 7efcb225f700 1 -- 172.21.15.88:0/2592317775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcac0ffc40 msgr2=0x7efcac1959e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.453+0000 7efcb225f700 1 --2- 172.21.15.88:0/2592317775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcac0ffc40 0x7efcac1959e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00d980 tx=0x7efc9c00dd40 comp rx=0 tx=0).stop 2023-07-14T17:42:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.453+0000 7efcb225f700 1 -- 172.21.15.88:0/2592317775 shutdown_connections 2023-07-14T17:42:58.436 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.453+0000 7efcb225f700 1 --2- 172.21.15.88:0/2592317775 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7efcac1017c0 0x7efcac19cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.453+0000 7efcb225f700 1 --2- 172.21.15.88:0/2592317775 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7efc940a1150 0x7efc940a3600 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.453+0000 7efcb225f700 1 --2- 172.21.15.88:0/2592317775 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efcac0ffc40 0x7efcac1959e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.453+0000 7efcb225f700 1 --2- 172.21.15.88:0/2592317775 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7efcac100e30 0x7efcac195f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.453+0000 7efcb225f700 1 -- 172.21.15.88:0/2592317775 >> 172.21.15.88:0/2592317775 conn(0x7efcac0fb240 msgr2=0x7efcac1042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:58.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.453+0000 7efcb225f700 1 -- 172.21.15.88:0/2592317775 shutdown_connections 2023-07-14T17:42:58.437 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.453+0000 7efcb225f700 1 -- 172.21.15.88:0/2592317775 wait complete. 2023-07-14T17:42:58.438 INFO:teuthology.orchestra.run.smithi088.stderr:listed 10 entries 2023-07-14T17:42:58.447 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi088.front.sepia.ceph.com... 2023-07-14T17:42:58.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:42:58.449 DEBUG:teuthology.orchestra.run.smithi088:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-07-14T17:42:58.487 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi088.stderr:ceph-fuse[164682]: fuse finished with error 0 and tester_r 0 2023-07-14T17:42:58.488 INFO:teuthology.orchestra.run:waiting for 300 2023-07-14T17:42:58.550 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2023-07-14T17:42:58.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:42:58.551 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-07-14T17:42:58.586 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:42:58.746 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.764+0000 7fa724f13700 1 -- 172.21.15.88:0/3925012503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7200ff750 msgr2=0x7fa720107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:58.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.764+0000 7fa724f13700 1 --2- 172.21.15.88:0/3925012503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7200ff750 0x7fa720107020 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa71800b0a0 tx=0x7fa71800b3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.764+0000 7fa724f13700 1 -- 172.21.15.88:0/3925012503 shutdown_connections 2023-07-14T17:42:58.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.764+0000 7fa724f13700 1 --2- 172.21.15.88:0/3925012503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7200ff750 0x7fa720107020 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.764+0000 7fa724f13700 1 --2- 172.21.15.88:0/3925012503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa720100090 0x7fa7201004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.747 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.764+0000 7fa724f13700 1 --2- 172.21.15.88:0/3925012503 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7200fedc0 0x7fa7200ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.764+0000 7fa724f13700 1 -- 172.21.15.88:0/3925012503 >> 172.21.15.88:0/3925012503 conn(0x7fa7200f98a0 msgr2=0x7fa7200fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:58.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.764+0000 7fa724f13700 1 -- 172.21.15.88:0/3925012503 shutdown_connections 2023-07-14T17:42:58.748 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.764+0000 7fa724f13700 1 -- 172.21.15.88:0/3925012503 wait complete. 2023-07-14T17:42:58.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.765+0000 7fa724f13700 1 Processor -- start 2023-07-14T17:42:58.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.765+0000 7fa724f13700 1 -- start start 2023-07-14T17:42:58.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.765+0000 7fa724f13700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7200fedc0 0x7fa720194790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:58.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.765+0000 7fa724f13700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7200ff750 0x7fa720194cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:58.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.765+0000 7fa724f13700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa720100090 0x7fa72019bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:58.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.765+0000 7fa724f13700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72019c310 con 0x7fa7200fedc0 2023-07-14T17:42:58.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.765+0000 7fa724f13700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72019c450 con 0x7fa720100090 2023-07-14T17:42:58.750 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.765+0000 7fa724f13700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72019c5e0 con 0x7fa7200ff750 2023-07-14T17:42:58.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.765+0000 7fa71e59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7200fedc0 0x7fa720194790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:58.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.765+0000 7fa71e59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7200fedc0 0x7fa720194790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41812/0 (socket says 172.21.15.88:41812) 2023-07-14T17:42:58.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.765+0000 7fa71e59c700 1 -- 172.21.15.88:0/4136971635 learned_addr learned my addr 172.21.15.88:0/4136971635 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:58.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.765+0000 7fa71ed9d700 1 --2- 172.21.15.88:0/4136971635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa720100090 0x7fa72019bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:58.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.765+0000 7fa71e59c700 1 -- 172.21.15.88:0/4136971635 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7200ff750 msgr2=0x7fa720194cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:58.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.765+0000 7fa71e59c700 1 --2- 172.21.15.88:0/4136971635 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7200ff750 0x7fa720194cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.765+0000 7fa71e59c700 1 -- 172.21.15.88:0/4136971635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa720100090 msgr2=0x7fa72019bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:58.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.765+0000 7fa71e59c700 1 --2- 172.21.15.88:0/4136971635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa720100090 0x7fa72019bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.765+0000 7fa71e59c700 1 -- 172.21.15.88:0/4136971635 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa718009d00 con 0x7fa7200fedc0 2023-07-14T17:42:58.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.766+0000 7fa71ed9d700 1 --2- 172.21.15.88:0/4136971635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa720100090 0x7fa72019bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:42:58.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.766+0000 7fa71e59c700 1 --2- 172.21.15.88:0/4136971635 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7200fedc0 0x7fa720194790 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa71400c990 tx=0x7fa71400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:58.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.766+0000 7fa70f7fe700 1 -- 172.21.15.88:0/4136971635 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa714007b00 con 0x7fa7200fedc0 2023-07-14T17:42:58.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.766+0000 7fa70f7fe700 1 -- 172.21.15.88:0/4136971635 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa714007c60 con 0x7fa7200fedc0 2023-07-14T17:42:58.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.766+0000 7fa70f7fe700 1 -- 172.21.15.88:0/4136971635 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa714018660 con 0x7fa7200fedc0 2023-07-14T17:42:58.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.766+0000 7fa724f13700 1 -- 172.21.15.88:0/4136971635 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa72019c8c0 con 0x7fa7200fedc0 2023-07-14T17:42:58.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.766+0000 7fa724f13700 1 -- 172.21.15.88:0/4136971635 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa72019cd80 con 0x7fa7200fedc0 2023-07-14T17:42:58.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.767+0000 7fa724f13700 1 -- 172.21.15.88:0/4136971635 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa72010a180 con 0x7fa7200fedc0 2023-07-14T17:42:58.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.768+0000 7fa70f7fe700 1 -- 172.21.15.88:0/4136971635 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa71400f450 con 0x7fa7200fedc0 2023-07-14T17:42:58.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.768+0000 7fa70f7fe700 1 --2- 172.21.15.88:0/4136971635 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa7080a11a0 0x7fa7080a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:58.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.768+0000 7fa71dd9b700 1 --2- 172.21.15.88:0/4136971635 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa7080a11a0 0x7fa7080a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:58.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.769+0000 7fa70f7fe700 1 -- 172.21.15.88:0/4136971635 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa7140ccb90 con 0x7fa7200fedc0 2023-07-14T17:42:58.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.769+0000 7fa71dd9b700 1 --2- 172.21.15.88:0/4136971635 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa7080a11a0 0x7fa7080a3650 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa710005950 tx=0x7fa7100058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:58.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.772+0000 7fa70f7fe700 1 -- 172.21.15.88:0/4136971635 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa714096180 con 0x7fa7200fedc0 2023-07-14T17:42:58.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.964+0000 7fa724f13700 1 -- 172.21.15.88:0/4136971635 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa72010a3a0 con 0x7fa7200fedc0 2023-07-14T17:42:58.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.965+0000 7fa70f7fe700 1 -- 172.21.15.88:0/4136971635 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23985 (secure 0 0 0) 0x7fa7140997a0 con 0x7fa7200fedc0 2023-07-14T17:42:58.948 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:58.948 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":86,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:42:46.934364+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:42:58.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.967+0000 7fa724f13700 1 -- 172.21.15.88:0/4136971635 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa7080a11a0 msgr2=0x7fa7080a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:58.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.967+0000 7fa724f13700 1 --2- 172.21.15.88:0/4136971635 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa7080a11a0 0x7fa7080a3650 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa710005950 tx=0x7fa7100058e0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.967+0000 7fa724f13700 1 -- 172.21.15.88:0/4136971635 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7200fedc0 msgr2=0x7fa720194790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:58.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.967+0000 7fa724f13700 1 --2- 172.21.15.88:0/4136971635 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7200fedc0 0x7fa720194790 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa71400c990 tx=0x7fa71400cd50 comp rx=0 tx=0).stop 2023-07-14T17:42:58.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.967+0000 7fa724f13700 1 -- 172.21.15.88:0/4136971635 shutdown_connections 2023-07-14T17:42:58.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.967+0000 7fa724f13700 1 --2- 172.21.15.88:0/4136971635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa720100090 0x7fa72019bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.967+0000 7fa724f13700 1 --2- 172.21.15.88:0/4136971635 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa7080a11a0 0x7fa7080a3650 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.967+0000 7fa724f13700 1 --2- 172.21.15.88:0/4136971635 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa7200fedc0 0x7fa720194790 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.967+0000 7fa724f13700 1 --2- 172.21.15.88:0/4136971635 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa7200ff750 0x7fa720194cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:58.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.967+0000 7fa724f13700 1 -- 172.21.15.88:0/4136971635 >> 172.21.15.88:0/4136971635 conn(0x7fa7200f98a0 msgr2=0x7fa7200fba20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:58.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.967+0000 7fa724f13700 1 -- 172.21.15.88:0/4136971635 shutdown_connections 2023-07-14T17:42:58.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:58.967+0000 7fa724f13700 1 -- 172.21.15.88:0/4136971635 wait complete. 2023-07-14T17:42:58.962 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:42:59.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.100+0000 7f0f0d98d700 1 -- 172.21.15.88:0/2705317102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f080fdf30 msgr2=0x7f0f080fe340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.100+0000 7f0f0d98d700 1 --2- 172.21.15.88:0/2705317102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f080fdf30 0x7f0f080fe340 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0efc009990 tx=0x7f0efc009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.100+0000 7f0f0d98d700 1 -- 172.21.15.88:0/2705317102 shutdown_connections 2023-07-14T17:42:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.100+0000 7f0f0d98d700 1 --2- 172.21.15.88:0/2705317102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f08101930 0x7f0f081081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.100+0000 7f0f0d98d700 1 --2- 172.21.15.88:0/2705317102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f080fdf30 0x7f0f080fe340 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.100+0000 7f0f0d98d700 1 --2- 172.21.15.88:0/2705317102 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f08100fa0 0x7f0f081013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.100+0000 7f0f0d98d700 1 -- 172.21.15.88:0/2705317102 >> 172.21.15.88:0/2705317102 conn(0x7f0f080f98d0 msgr2=0x7f0f080fbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:59.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.100+0000 7f0f0d98d700 1 -- 172.21.15.88:0/2705317102 shutdown_connections 2023-07-14T17:42:59.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.100+0000 7f0f0d98d700 1 -- 172.21.15.88:0/2705317102 wait complete. 2023-07-14T17:42:59.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.101+0000 7f0f0d98d700 1 Processor -- start 2023-07-14T17:42:59.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.101+0000 7f0f0d98d700 1 -- start start 2023-07-14T17:42:59.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.101+0000 7f0f0d98d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f080fdf30 0x7f0f08100910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:59.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.101+0000 7f0f0d98d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f08100fa0 0x7f0f080fef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:59.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.101+0000 7f0f0d98d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f08101930 0x7f0f080ff490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:59.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.101+0000 7f0f0d98d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f080ffc30 con 0x7f0f08101930 2023-07-14T17:42:59.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.101+0000 7f0f0d98d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f080ffda0 con 0x7f0f080fdf30 2023-07-14T17:42:59.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.101+0000 7f0f0d98d700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0819f8b0 con 0x7f0f08100fa0 2023-07-14T17:42:59.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.101+0000 7f0f077fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f08101930 0x7f0f080ff490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:59.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.101+0000 7f0f077fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f08101930 0x7f0f080ff490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41830/0 (socket says 172.21.15.88:41830) 2023-07-14T17:42:59.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.101+0000 7f0f077fe700 1 -- 172.21.15.88:0/4215105897 learned_addr learned my addr 172.21.15.88:0/4215105897 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:59.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.101+0000 7f0f067fc700 1 --2- 172.21.15.88:0/4215105897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f08100fa0 0x7f0f080fef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:59.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.101+0000 7f0f06ffd700 1 --2- 172.21.15.88:0/4215105897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f080fdf30 0x7f0f08100910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:59.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.101+0000 7f0f077fe700 1 -- 172.21.15.88:0/4215105897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f08100fa0 msgr2=0x7f0f080fef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:59.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.101+0000 7f0f077fe700 1 --2- 172.21.15.88:0/4215105897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f08100fa0 0x7f0f080fef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.101+0000 7f0f077fe700 1 -- 172.21.15.88:0/4215105897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f080fdf30 msgr2=0x7f0f08100910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:59.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.101+0000 7f0f077fe700 1 --2- 172.21.15.88:0/4215105897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f080fdf30 0x7f0f08100910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.102+0000 7f0f077fe700 1 -- 172.21.15.88:0/4215105897 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0efc0095f0 con 0x7f0f08101930 2023-07-14T17:42:59.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.102+0000 7f0f06ffd700 1 --2- 172.21.15.88:0/4215105897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f080fdf30 0x7f0f08100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:42:59.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.102+0000 7f0f067fc700 1 --2- 172.21.15.88:0/4215105897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f08100fa0 0x7f0f080fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:42:59.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.102+0000 7f0f077fe700 1 --2- 172.21.15.88:0/4215105897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f08101930 0x7f0f080ff490 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0f00007ee0 tx=0x7f0f0000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:59.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.102+0000 7f0f0c98b700 1 -- 172.21.15.88:0/4215105897 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f00017070 con 0x7f0f08101930 2023-07-14T17:42:59.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.102+0000 7f0f0c98b700 1 -- 172.21.15.88:0/4215105897 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0f0000f040 con 0x7f0f08101930 2023-07-14T17:42:59.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.102+0000 7f0f0d98d700 1 -- 172.21.15.88:0/4215105897 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0819fb90 con 0x7f0f08101930 2023-07-14T17:42:59.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.102+0000 7f0f0c98b700 1 -- 172.21.15.88:0/4215105897 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f00005710 con 0x7f0f08101930 2023-07-14T17:42:59.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.102+0000 7f0f0d98d700 1 -- 172.21.15.88:0/4215105897 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f081a0050 con 0x7f0f08101930 2023-07-14T17:42:59.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.103+0000 7f0f0d98d700 1 -- 172.21.15.88:0/4215105897 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f081072f0 con 0x7f0f08101930 2023-07-14T17:42:59.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.104+0000 7f0f0c98b700 1 -- 172.21.15.88:0/4215105897 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0f00005870 con 0x7f0f08101930 2023-07-14T17:42:59.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.104+0000 7f0f0c98b700 1 --2- 172.21.15.88:0/4215105897 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0ef00a11a0 0x7f0ef00a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:59.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.105+0000 7f0f06ffd700 1 --2- 172.21.15.88:0/4215105897 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0ef00a11a0 0x7f0ef00a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:59.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.105+0000 7f0f0c98b700 1 -- 172.21.15.88:0/4215105897 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0f000cc190 con 0x7f0f08101930 2023-07-14T17:42:59.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.105+0000 7f0f06ffd700 1 --2- 172.21.15.88:0/4215105897 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0ef00a11a0 0x7f0ef00a3650 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0efc000c00 tx=0x7f0efc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:59.091 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.108+0000 7f0f0c98b700 1 -- 172.21.15.88:0/4215105897 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0f000956f0 con 0x7f0f08101930 2023-07-14T17:42:59.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:58 smithi088 ceph-mon[127465]: pgmap v263: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 297 B/s wr, 0 op/s 2023-07-14T17:42:59.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:58 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1764339983' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:42:59.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:58 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2592317775' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T17:42:59.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:58 smithi107 ceph-mon[128999]: pgmap v263: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 297 B/s wr, 0 op/s 2023-07-14T17:42:59.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:58 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1764339983' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:42:59.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:58 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2592317775' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T17:42:59.280 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.298+0000 7f0f0d98d700 1 -- 172.21.15.88:0/4215105897 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f0802cb30 con 0x7f0f08101930 2023-07-14T17:42:59.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.300+0000 7f0f0c98b700 1 -- 172.21.15.88:0/4215105897 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23985 (secure 0 0 0) 0x7f0f00098d10 con 0x7f0f08101930 2023-07-14T17:42:59.282 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:42:59.282 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":86,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:42:46.934364+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:42:59.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.302+0000 7f0f0d98d700 1 -- 172.21.15.88:0/4215105897 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0ef00a11a0 msgr2=0x7f0ef00a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:59.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.302+0000 7f0f0d98d700 1 --2- 172.21.15.88:0/4215105897 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0ef00a11a0 0x7f0ef00a3650 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0efc000c00 tx=0x7f0efc005fb0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.302+0000 7f0f0d98d700 1 -- 172.21.15.88:0/4215105897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f08101930 msgr2=0x7f0f080ff490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:59.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.302+0000 7f0f0d98d700 1 --2- 172.21.15.88:0/4215105897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f08101930 0x7f0f080ff490 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0f00007ee0 tx=0x7f0f0000d3b0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.302+0000 7f0f0d98d700 1 -- 172.21.15.88:0/4215105897 shutdown_connections 2023-07-14T17:42:59.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.302+0000 7f0f0d98d700 1 --2- 172.21.15.88:0/4215105897 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0ef00a11a0 0x7f0ef00a3650 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.302+0000 7f0f0d98d700 1 --2- 172.21.15.88:0/4215105897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f08101930 0x7f0f080ff490 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.302+0000 7f0f0d98d700 1 --2- 172.21.15.88:0/4215105897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f080fdf30 0x7f0f08100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.302+0000 7f0f0d98d700 1 --2- 172.21.15.88:0/4215105897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f08100fa0 0x7f0f080fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.302+0000 7f0f0d98d700 1 -- 172.21.15.88:0/4215105897 >> 172.21.15.88:0/4215105897 conn(0x7f0f080f98d0 msgr2=0x7f0f08104430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:59.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.302+0000 7f0f0d98d700 1 -- 172.21.15.88:0/4215105897 shutdown_connections 2023-07-14T17:42:59.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.302+0000 7f0f0d98d700 1 -- 172.21.15.88:0/4215105897 wait complete. 2023-07-14T17:42:59.296 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-07-14T17:42:59.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:58 smithi178 ceph-mon[129368]: pgmap v263: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 297 B/s wr, 0 op/s 2023-07-14T17:42:59.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:58 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1764339983' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:42:59.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:58 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2592317775' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T17:42:59.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.438+0000 7f3370287700 1 -- 172.21.15.88:0/3063003454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33681000f0 msgr2=0x7f3368100500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:59.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.438+0000 7f3370287700 1 --2- 172.21.15.88:0/3063003454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33681000f0 0x7f3368100500 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3358009990 tx=0x7f3358009ca0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.438+0000 7f3370287700 1 -- 172.21.15.88:0/3063003454 shutdown_connections 2023-07-14T17:42:59.424 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.438+0000 7f3370287700 1 --2- 172.21.15.88:0/3063003454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33680ff6f0 0x7f3368106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.438+0000 7f3370287700 1 --2- 172.21.15.88:0/3063003454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33681000f0 0x7f3368100500 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.438+0000 7f3370287700 1 --2- 172.21.15.88:0/3063003454 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f33680fed40 0x7f33680ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.438+0000 7f3370287700 1 -- 172.21.15.88:0/3063003454 >> 172.21.15.88:0/3063003454 conn(0x7f33680f98e0 msgr2=0x7f33680fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:59.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.438+0000 7f3370287700 1 -- 172.21.15.88:0/3063003454 shutdown_connections 2023-07-14T17:42:59.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.438+0000 7f3370287700 1 -- 172.21.15.88:0/3063003454 wait complete. 2023-07-14T17:42:59.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.438+0000 7f3370287700 1 Processor -- start 2023-07-14T17:42:59.425 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f3370287700 1 -- start start 2023-07-14T17:42:59.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f3370287700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33680fed40 0x7f3368197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:59.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f3370287700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33680ff6f0 0x7f3368198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:59.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f3370287700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f33681000f0 0x7f336819f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:59.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f3370287700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336819f640 con 0x7f33680fed40 2023-07-14T17:42:59.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f3370287700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336819f780 con 0x7f33680ff6f0 2023-07-14T17:42:59.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f3370287700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336819f910 con 0x7f33681000f0 2023-07-14T17:42:59.426 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f336e023700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33680fed40 0x7f3368197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:59.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f336e023700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33680fed40 0x7f3368197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41840/0 (socket says 172.21.15.88:41840) 2023-07-14T17:42:59.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f336e023700 1 -- 172.21.15.88:0/928810499 learned_addr learned my addr 172.21.15.88:0/928810499 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:42:59.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f336e824700 1 --2- 172.21.15.88:0/928810499 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f33681000f0 0x7f336819f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:59.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f336e023700 1 -- 172.21.15.88:0/928810499 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f33681000f0 msgr2=0x7f336819f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:59.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f336e023700 1 --2- 172.21.15.88:0/928810499 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f33681000f0 0x7f336819f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f336e023700 1 -- 172.21.15.88:0/928810499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33680ff6f0 msgr2=0x7f3368198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:59.427 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f336e023700 1 --2- 172.21.15.88:0/928810499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33680ff6f0 0x7f3368198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f336e023700 1 -- 172.21.15.88:0/928810499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33580095f0 con 0x7f33680fed40 2023-07-14T17:42:59.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f336e023700 1 --2- 172.21.15.88:0/928810499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33680fed40 0x7f3368197ac0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3358006010 tx=0x7f3358004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:59.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f335f7fe700 1 -- 172.21.15.88:0/928810499 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335801d070 con 0x7f33680fed40 2023-07-14T17:42:59.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f335f7fe700 1 -- 172.21.15.88:0/928810499 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f33580041b0 con 0x7f33680fed40 2023-07-14T17:42:59.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f3370287700 1 -- 172.21.15.88:0/928810499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336819fb90 con 0x7f33680fed40 2023-07-14T17:42:59.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f335f7fe700 1 -- 172.21.15.88:0/928810499 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358022da0 con 0x7f33680fed40 2023-07-14T17:42:59.428 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.439+0000 7f3370287700 1 -- 172.21.15.88:0/928810499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336819ffb0 con 0x7f33680fed40 2023-07-14T17:42:59.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.440+0000 7f3370287700 1 -- 172.21.15.88:0/928810499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f336810b310 con 0x7f33680fed40 2023-07-14T17:42:59.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.441+0000 7f335f7fe700 1 -- 172.21.15.88:0/928810499 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3358022570 con 0x7f33680fed40 2023-07-14T17:42:59.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.442+0000 7f335f7fe700 1 --2- 172.21.15.88:0/928810499 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f33540a1150 0x7f33540a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:42:59.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.442+0000 7f335f7fe700 1 -- 172.21.15.88:0/928810499 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f33580ce0e0 con 0x7f33680fed40 2023-07-14T17:42:59.429 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.442+0000 7f336d822700 1 --2- 172.21.15.88:0/928810499 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f33540a1150 0x7f33540a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:42:59.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.442+0000 7f336d822700 1 --2- 172.21.15.88:0/928810499 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f33540a1150 0x7f33540a3600 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f336400a990 tx=0x7f3364005c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:42:59.430 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.446+0000 7f335f7fe700 1 -- 172.21.15.88:0/928810499 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3358097670 con 0x7f33680fed40 2023-07-14T17:42:59.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.677+0000 7f3370287700 1 -- 172.21.15.88:0/928810499 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f33681a05c0 con 0x7f33680fed40 2023-07-14T17:42:59.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.945+0000 7f335f7fe700 1 -- 172.21.15.88:0/928810499 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f335809ac90 con 0x7f33680fed40 2023-07-14T17:42:59.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.947+0000 7f3370287700 1 -- 172.21.15.88:0/928810499 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f33540a1150 msgr2=0x7f33540a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:59.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.947+0000 7f3370287700 1 --2- 172.21.15.88:0/928810499 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f33540a1150 0x7f33540a3600 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f336400a990 tx=0x7f3364005c80 comp rx=0 tx=0).stop 2023-07-14T17:42:59.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.947+0000 7f3370287700 1 -- 172.21.15.88:0/928810499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33680fed40 msgr2=0x7f3368197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:42:59.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.947+0000 7f3370287700 1 --2- 172.21.15.88:0/928810499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33680fed40 0x7f3368197ac0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3358006010 tx=0x7f3358004730 comp rx=0 tx=0).stop 2023-07-14T17:42:59.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.947+0000 7f3370287700 1 -- 172.21.15.88:0/928810499 shutdown_connections 2023-07-14T17:42:59.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.947+0000 7f3370287700 1 --2- 172.21.15.88:0/928810499 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f33681000f0 0x7f336819f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.947+0000 7f3370287700 1 --2- 172.21.15.88:0/928810499 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f33540a1150 0x7f33540a3600 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.947+0000 7f3370287700 1 --2- 172.21.15.88:0/928810499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f33680fed40 0x7f3368197ac0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.947+0000 7f3370287700 1 --2- 172.21.15.88:0/928810499 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f33680ff6f0 0x7f3368198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:42:59.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.947+0000 7f3370287700 1 -- 172.21.15.88:0/928810499 >> 172.21.15.88:0/928810499 conn(0x7f33680f98e0 msgr2=0x7f33680fbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:42:59.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.947+0000 7f3370287700 1 -- 172.21.15.88:0/928810499 shutdown_connections 2023-07-14T17:42:59.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:42:59.947+0000 7f3370287700 1 -- 172.21.15.88:0/928810499 wait complete. 2023-07-14T17:42:59.941 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2023-07-14T17:42:59.942 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2023-07-14T17:43:00.067 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.083+0000 7f147adb9700 1 -- 172.21.15.88:0/3373446239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14740ffd00 msgr2=0x7f1474100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:00.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.083+0000 7f147adb9700 1 --2- 172.21.15.88:0/3373446239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14740ffd00 0x7f1474100150 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1460009990 tx=0x7f1460009ca0 comp rx=0 tx=0).stop 2023-07-14T17:43:00.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.084+0000 7f147adb9700 1 -- 172.21.15.88:0/3373446239 shutdown_connections 2023-07-14T17:43:00.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.084+0000 7f147adb9700 1 --2- 172.21.15.88:0/3373446239 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1474100690 0x7f1474106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:00.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.084+0000 7f147adb9700 1 --2- 172.21.15.88:0/3373446239 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f14740feb10 0x7f14740fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:00.068 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.084+0000 7f147adb9700 1 --2- 172.21.15.88:0/3373446239 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14740ffd00 0x7f1474100150 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:00.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.084+0000 7f147adb9700 1 -- 172.21.15.88:0/3373446239 >> 172.21.15.88:0/3373446239 conn(0x7f14740fa130 msgr2=0x7f14740fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:00.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.084+0000 7f147adb9700 1 -- 172.21.15.88:0/3373446239 shutdown_connections 2023-07-14T17:43:00.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.084+0000 7f147adb9700 1 -- 172.21.15.88:0/3373446239 wait complete. 2023-07-14T17:43:00.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.084+0000 7f147adb9700 1 Processor -- start 2023-07-14T17:43:00.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.084+0000 7f147adb9700 1 -- start start 2023-07-14T17:43:00.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.084+0000 7f147adb9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14740feb10 0x7f1474068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:00.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.084+0000 7f147adb9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14740ffd00 0x7f147419a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:00.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.084+0000 7f147adb9700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1474100690 0x7f147419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:00.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.084+0000 7f147adb9700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147419d410 con 0x7f14740feb10 2023-07-14T17:43:00.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.084+0000 7f147adb9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147419d580 con 0x7f14740ffd00 2023-07-14T17:43:00.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.084+0000 7f147adb9700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147419d740 con 0x7f1474100690 2023-07-14T17:43:00.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.084+0000 7f1478b55700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14740feb10 0x7f1474068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:00.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.085+0000 7f1478b55700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14740feb10 0x7f1474068330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41854/0 (socket says 172.21.15.88:41854) 2023-07-14T17:43:00.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.085+0000 7f1478b55700 1 -- 172.21.15.88:0/4280231766 learned_addr learned my addr 172.21.15.88:0/4280231766 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:00.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.085+0000 7f1473fff700 1 --2- 172.21.15.88:0/4280231766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14740ffd00 0x7f147419a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:00.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.085+0000 7f1479356700 1 --2- 172.21.15.88:0/4280231766 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1474100690 0x7f147419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:00.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.085+0000 7f1478b55700 1 -- 172.21.15.88:0/4280231766 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1474100690 msgr2=0x7f147419cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:00.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.085+0000 7f1478b55700 1 --2- 172.21.15.88:0/4280231766 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1474100690 0x7f147419cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:00.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.085+0000 7f1478b55700 1 -- 172.21.15.88:0/4280231766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14740ffd00 msgr2=0x7f147419a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:00.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.085+0000 7f1478b55700 1 --2- 172.21.15.88:0/4280231766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14740ffd00 0x7f147419a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:00.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.085+0000 7f1478b55700 1 -- 172.21.15.88:0/4280231766 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14600095f0 con 0x7f14740feb10 2023-07-14T17:43:00.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.085+0000 7f1479356700 1 --2- 172.21.15.88:0/4280231766 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1474100690 0x7f147419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:43:00.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.085+0000 7f1478b55700 1 --2- 172.21.15.88:0/4280231766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14740feb10 0x7f1474068330 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f146800c970 tx=0x7f146800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:00.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.085+0000 7f1471ffb700 1 -- 172.21.15.88:0/4280231766 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468004220 con 0x7f14740feb10 2023-07-14T17:43:00.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.085+0000 7f1471ffb700 1 -- 172.21.15.88:0/4280231766 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1468007d30 con 0x7f14740feb10 2023-07-14T17:43:00.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.085+0000 7f147adb9700 1 -- 172.21.15.88:0/4280231766 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147419da20 con 0x7f14740feb10 2023-07-14T17:43:00.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.085+0000 7f1471ffb700 1 -- 172.21.15.88:0/4280231766 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468003b80 con 0x7f14740feb10 2023-07-14T17:43:00.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.085+0000 7f147adb9700 1 -- 172.21.15.88:0/4280231766 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147419df00 con 0x7f14740feb10 2023-07-14T17:43:00.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.086+0000 7f147adb9700 1 -- 172.21.15.88:0/4280231766 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f147410a190 con 0x7f14740feb10 2023-07-14T17:43:00.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.087+0000 7f1471ffb700 1 -- 172.21.15.88:0/4280231766 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1468003ce0 con 0x7f14740feb10 2023-07-14T17:43:00.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.088+0000 7f1471ffb700 1 --2- 172.21.15.88:0/4280231766 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f14640a11d0 0x7f14640a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:00.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.088+0000 7f1473fff700 1 --2- 172.21.15.88:0/4280231766 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f14640a11d0 0x7f14640a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:00.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.088+0000 7f1471ffb700 1 -- 172.21.15.88:0/4280231766 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f14680cc790 con 0x7f14740feb10 2023-07-14T17:43:00.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.088+0000 7f1473fff700 1 --2- 172.21.15.88:0/4280231766 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f14640a11d0 0x7f14640a3680 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f146000c010 tx=0x7f1460005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:00.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.092+0000 7f1471ffb700 1 -- 172.21.15.88:0/4280231766 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1468095c90 con 0x7f14740feb10 2023-07-14T17:43:00.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:59 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/4136971635' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:43:00.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:59 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/4215105897' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:43:00.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:42:59 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/928810499' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T17:43:00.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:59 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/4136971635' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:43:00.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:59 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/4215105897' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:43:00.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:42:59 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/928810499' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T17:43:00.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.324+0000 7f147adb9700 1 -- 172.21.15.88:0/4280231766 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f1474105b10 con 0x7f14740feb10 2023-07-14T17:43:00.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:59 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/4136971635' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:43:00.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:59 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/4215105897' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:43:00.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:42:59 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/928810499' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T17:43:00.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.734+0000 7f1471ffb700 1 -- 172.21.15.88:0/4280231766 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7f14680992b0 con 0x7f14740feb10 2023-07-14T17:43:00.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.737+0000 7f147adb9700 1 -- 172.21.15.88:0/4280231766 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f14640a11d0 msgr2=0x7f14640a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:00.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.737+0000 7f147adb9700 1 --2- 172.21.15.88:0/4280231766 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f14640a11d0 0x7f14640a3680 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f146000c010 tx=0x7f1460005960 comp rx=0 tx=0).stop 2023-07-14T17:43:00.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.737+0000 7f147adb9700 1 -- 172.21.15.88:0/4280231766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14740feb10 msgr2=0x7f1474068330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:00.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.737+0000 7f147adb9700 1 --2- 172.21.15.88:0/4280231766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14740feb10 0x7f1474068330 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f146800c970 tx=0x7f146800cc80 comp rx=0 tx=0).stop 2023-07-14T17:43:00.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.737+0000 7f147adb9700 1 -- 172.21.15.88:0/4280231766 shutdown_connections 2023-07-14T17:43:00.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.737+0000 7f147adb9700 1 --2- 172.21.15.88:0/4280231766 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1474100690 0x7f147419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:00.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.737+0000 7f147adb9700 1 --2- 172.21.15.88:0/4280231766 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f14640a11d0 0x7f14640a3680 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:00.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.737+0000 7f147adb9700 1 --2- 172.21.15.88:0/4280231766 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f14740feb10 0x7f1474068330 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:00.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.737+0000 7f147adb9700 1 --2- 172.21.15.88:0/4280231766 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f14740ffd00 0x7f147419a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:00.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.737+0000 7f147adb9700 1 -- 172.21.15.88:0/4280231766 >> 172.21.15.88:0/4280231766 conn(0x7f14740fa130 msgr2=0x7f14740fc460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:00.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.737+0000 7f147adb9700 1 -- 172.21.15.88:0/4280231766 shutdown_connections 2023-07-14T17:43:00.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.737+0000 7f147adb9700 1 -- 172.21.15.88:0/4280231766 wait complete. 2023-07-14T17:43:00.736 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-07-14T17:43:00.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.879+0000 7ff46046b700 1 -- 172.21.15.88:0/2328043395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4580ffd00 msgr2=0x7ff458100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:00.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.879+0000 7ff46046b700 1 --2- 172.21.15.88:0/2328043395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4580ffd00 0x7ff458100150 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff448009990 tx=0x7ff448009ca0 comp rx=0 tx=0).stop 2023-07-14T17:43:00.862 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.880+0000 7ff46046b700 1 -- 172.21.15.88:0/2328043395 shutdown_connections 2023-07-14T17:43:00.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.880+0000 7ff46046b700 1 --2- 172.21.15.88:0/2328043395 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff458100690 0x7ff458106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:00.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.880+0000 7ff46046b700 1 --2- 172.21.15.88:0/2328043395 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4580feb10 0x7ff4580fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:00.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.880+0000 7ff46046b700 1 --2- 172.21.15.88:0/2328043395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4580ffd00 0x7ff458100150 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:00.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.880+0000 7ff46046b700 1 -- 172.21.15.88:0/2328043395 >> 172.21.15.88:0/2328043395 conn(0x7ff4580fa130 msgr2=0x7ff4580fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:00.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.880+0000 7ff46046b700 1 -- 172.21.15.88:0/2328043395 shutdown_connections 2023-07-14T17:43:00.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.880+0000 7ff46046b700 1 -- 172.21.15.88:0/2328043395 wait complete. 2023-07-14T17:43:00.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.880+0000 7ff46046b700 1 Processor -- start 2023-07-14T17:43:00.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.880+0000 7ff46046b700 1 -- start start 2023-07-14T17:43:00.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.880+0000 7ff46046b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4580feb10 0x7ff458068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:00.865 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.880+0000 7ff46046b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4580ffd00 0x7ff45819a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:00.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.880+0000 7ff46046b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff458100690 0x7ff45819cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:00.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.880+0000 7ff46046b700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45819d380 con 0x7ff458100690 2023-07-14T17:43:00.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.880+0000 7ff46046b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45819d4f0 con 0x7ff4580ffd00 2023-07-14T17:43:00.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.880+0000 7ff46046b700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45819d6b0 con 0x7ff4580feb10 2023-07-14T17:43:00.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.880+0000 7ff45ea08700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff458100690 0x7ff45819cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:00.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.881+0000 7ff45ea08700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff458100690 0x7ff45819cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41872/0 (socket says 172.21.15.88:41872) 2023-07-14T17:43:00.866 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.881+0000 7ff45ea08700 1 -- 172.21.15.88:0/2639073105 learned_addr learned my addr 172.21.15.88:0/2639073105 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:00.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.881+0000 7ff45da06700 1 --2- 172.21.15.88:0/2639073105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4580ffd00 0x7ff45819a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:00.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.881+0000 7ff45e207700 1 --2- 172.21.15.88:0/2639073105 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4580feb10 0x7ff458068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:00.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.881+0000 7ff45ea08700 1 -- 172.21.15.88:0/2639073105 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4580feb10 msgr2=0x7ff458068330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:00.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.881+0000 7ff45ea08700 1 --2- 172.21.15.88:0/2639073105 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4580feb10 0x7ff458068330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:00.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.881+0000 7ff45ea08700 1 -- 172.21.15.88:0/2639073105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4580ffd00 msgr2=0x7ff45819a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:00.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.881+0000 7ff45ea08700 1 --2- 172.21.15.88:0/2639073105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4580ffd00 0x7ff45819a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:00.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.881+0000 7ff45ea08700 1 -- 172.21.15.88:0/2639073105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4480095f0 con 0x7ff458100690 2023-07-14T17:43:00.867 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.881+0000 7ff45ea08700 1 --2- 172.21.15.88:0/2639073105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff458100690 0x7ff45819cbb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff45000ada0 tx=0x7ff45000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:00.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.881+0000 7ff44f7fe700 1 -- 172.21.15.88:0/2639073105 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45000f070 con 0x7ff458100690 2023-07-14T17:43:00.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.881+0000 7ff44f7fe700 1 -- 172.21.15.88:0/2639073105 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff450007b30 con 0x7ff458100690 2023-07-14T17:43:00.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.881+0000 7ff46046b700 1 -- 172.21.15.88:0/2639073105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45819d990 con 0x7ff458100690 2023-07-14T17:43:00.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.881+0000 7ff44f7fe700 1 -- 172.21.15.88:0/2639073105 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4500053e0 con 0x7ff458100690 2023-07-14T17:43:00.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.881+0000 7ff46046b700 1 -- 172.21.15.88:0/2639073105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45819dea0 con 0x7ff458100690 2023-07-14T17:43:00.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.882+0000 7ff46046b700 1 -- 172.21.15.88:0/2639073105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4581060f0 con 0x7ff458100690 2023-07-14T17:43:00.868 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.883+0000 7ff44f7fe700 1 -- 172.21.15.88:0/2639073105 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff45000c980 con 0x7ff458100690 2023-07-14T17:43:00.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.884+0000 7ff44f7fe700 1 --2- 172.21.15.88:0/2639073105 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff4440a1250 0x7ff4440a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:00.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.884+0000 7ff45e207700 1 --2- 172.21.15.88:0/2639073105 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff4440a1250 0x7ff4440a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:00.869 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.884+0000 7ff44f7fe700 1 -- 172.21.15.88:0/2639073105 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff4500cc840 con 0x7ff458100690 2023-07-14T17:43:00.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.884+0000 7ff45e207700 1 --2- 172.21.15.88:0/2639073105 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff4440a1250 0x7ff4440a3700 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff4580ffb60 tx=0x7ff454005d70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:00.870 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:00.887+0000 7ff44f7fe700 1 -- 172.21.15.88:0/2639073105 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff450095dd0 con 0x7ff458100690 2023-07-14T17:43:01.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.120+0000 7ff46046b700 1 -- 172.21.15.88:0/2639073105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7ff45802c8e0 con 0x7ff458100690 2023-07-14T17:43:01.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:00 smithi088 ceph-mon[127465]: pgmap v264: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2023-07-14T17:43:01.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:00 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/928810499' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T17:43:01.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:00 smithi088 ceph-mon[127465]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T17:43:01.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:00 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/4280231766' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-14T17:43:01.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:00 smithi088 ceph-mon[127465]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T17:43:01.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:00 smithi088 ceph-mon[127465]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T17:43:01.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:00 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/4280231766' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-14T17:43:01.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:00 smithi088 ceph-mon[127465]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2023-07-14T17:43:01.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:00 smithi088 ceph-mon[127465]: daemon mds.h is now active in filesystem cephfs as rank 1 2023-07-14T17:43:01.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:00 smithi107 ceph-mon[128999]: pgmap v264: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2023-07-14T17:43:01.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:00 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/928810499' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T17:43:01.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:00 smithi107 ceph-mon[128999]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T17:43:01.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:00 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/4280231766' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-14T17:43:01.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:00 smithi107 ceph-mon[128999]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T17:43:01.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:00 smithi107 ceph-mon[128999]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T17:43:01.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:00 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/4280231766' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-14T17:43:01.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:00 smithi107 ceph-mon[128999]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2023-07-14T17:43:01.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:00 smithi107 ceph-mon[128999]: daemon mds.h is now active in filesystem cephfs as rank 1 2023-07-14T17:43:01.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:00 smithi178 ceph-mon[129368]: pgmap v264: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2023-07-14T17:43:01.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:00 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/928810499' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T17:43:01.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:00 smithi178 ceph-mon[129368]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T17:43:01.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:00 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/4280231766' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-14T17:43:01.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:00 smithi178 ceph-mon[129368]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T17:43:01.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:00 smithi178 ceph-mon[129368]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T17:43:01.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:00 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/4280231766' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-14T17:43:01.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:00 smithi178 ceph-mon[129368]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2023-07-14T17:43:01.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:00 smithi178 ceph-mon[129368]: daemon mds.h is now active in filesystem cephfs as rank 1 2023-07-14T17:43:01.723 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.741+0000 7ff44f7fe700 1 -- 172.21.15.88:0/2639073105 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7ff45000cc30 con 0x7ff458100690 2023-07-14T17:43:01.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.744+0000 7ff46046b700 1 -- 172.21.15.88:0/2639073105 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff4440a1250 msgr2=0x7ff4440a3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:01.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.744+0000 7ff46046b700 1 --2- 172.21.15.88:0/2639073105 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff4440a1250 0x7ff4440a3700 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff4580ffb60 tx=0x7ff454005d70 comp rx=0 tx=0).stop 2023-07-14T17:43:01.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.744+0000 7ff46046b700 1 -- 172.21.15.88:0/2639073105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff458100690 msgr2=0x7ff45819cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:01.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.744+0000 7ff46046b700 1 --2- 172.21.15.88:0/2639073105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff458100690 0x7ff45819cbb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff45000ada0 tx=0x7ff45000c330 comp rx=0 tx=0).stop 2023-07-14T17:43:01.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.744+0000 7ff46046b700 1 -- 172.21.15.88:0/2639073105 shutdown_connections 2023-07-14T17:43:01.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.744+0000 7ff46046b700 1 --2- 172.21.15.88:0/2639073105 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff4440a1250 0x7ff4440a3700 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:01.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.744+0000 7ff46046b700 1 --2- 172.21.15.88:0/2639073105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff458100690 0x7ff45819cbb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:01.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.744+0000 7ff46046b700 1 --2- 172.21.15.88:0/2639073105 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff4580feb10 0x7ff458068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:01.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.744+0000 7ff46046b700 1 --2- 172.21.15.88:0/2639073105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff4580ffd00 0x7ff45819a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:01.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.744+0000 7ff46046b700 1 -- 172.21.15.88:0/2639073105 >> 172.21.15.88:0/2639073105 conn(0x7ff4580fa130 msgr2=0x7ff458103230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:01.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.744+0000 7ff46046b700 1 -- 172.21.15.88:0/2639073105 shutdown_connections 2023-07-14T17:43:01.728 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.744+0000 7ff46046b700 1 -- 172.21.15.88:0/2639073105 wait complete. 2023-07-14T17:43:01.744 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-07-14T17:43:01.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.891+0000 7fbc258ab700 1 -- 172.21.15.88:0/4013071819 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc201006d0 msgr2=0x7fbc20106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:01.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.891+0000 7fbc258ab700 1 --2- 172.21.15.88:0/4013071819 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc201006d0 0x7fbc20106f90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800b0a0 tx=0x7fbc1800b3b0 comp rx=0 tx=0).stop 2023-07-14T17:43:01.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.891+0000 7fbc258ab700 1 -- 172.21.15.88:0/4013071819 shutdown_connections 2023-07-14T17:43:01.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.891+0000 7fbc258ab700 1 --2- 172.21.15.88:0/4013071819 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc201006d0 0x7fbc20106f90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:01.874 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.891+0000 7fbc258ab700 1 --2- 172.21.15.88:0/4013071819 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc200feb50 0x7fbc200fef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:01.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.891+0000 7fbc258ab700 1 --2- 172.21.15.88:0/4013071819 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc200ffd40 0x7fbc20100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:01.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.891+0000 7fbc258ab700 1 -- 172.21.15.88:0/4013071819 >> 172.21.15.88:0/4013071819 conn(0x7fbc200fa150 msgr2=0x7fbc200fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:01.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.891+0000 7fbc258ab700 1 -- 172.21.15.88:0/4013071819 shutdown_connections 2023-07-14T17:43:01.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.891+0000 7fbc258ab700 1 -- 172.21.15.88:0/4013071819 wait complete. 2023-07-14T17:43:01.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc258ab700 1 Processor -- start 2023-07-14T17:43:01.875 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc258ab700 1 -- start start 2023-07-14T17:43:01.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc258ab700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc200feb50 0x7fbc20069500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:01.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc258ab700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc200ffd40 0x7fbc2019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:01.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc258ab700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc201006d0 0x7fbc2019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:01.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc258ab700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2019d480 con 0x7fbc200feb50 2023-07-14T17:43:01.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc258ab700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2019d5f0 con 0x7fbc200ffd40 2023-07-14T17:43:01.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc258ab700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2019d7b0 con 0x7fbc201006d0 2023-07-14T17:43:01.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc1effd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc200feb50 0x7fbc20069500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:01.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc1effd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc200feb50 0x7fbc20069500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41882/0 (socket says 172.21.15.88:41882) 2023-07-14T17:43:01.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc1effd700 1 -- 172.21.15.88:0/407308505 learned_addr learned my addr 172.21.15.88:0/407308505 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:01.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc1e7fc700 1 --2- 172.21.15.88:0/407308505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc200ffd40 0x7fbc2019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:01.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc1f7fe700 1 --2- 172.21.15.88:0/407308505 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc201006d0 0x7fbc2019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:01.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc1effd700 1 -- 172.21.15.88:0/407308505 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc201006d0 msgr2=0x7fbc2019cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:01.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc1effd700 1 --2- 172.21.15.88:0/407308505 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc201006d0 0x7fbc2019cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:01.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc1effd700 1 -- 172.21.15.88:0/407308505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc200ffd40 msgr2=0x7fbc2019a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:01.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc1effd700 1 --2- 172.21.15.88:0/407308505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc200ffd40 0x7fbc2019a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:01.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc1effd700 1 -- 172.21.15.88:0/407308505 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc18009d00 con 0x7fbc200feb50 2023-07-14T17:43:01.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.892+0000 7fbc1e7fc700 1 --2- 172.21.15.88:0/407308505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc200ffd40 0x7fbc2019a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:43:01.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.893+0000 7fbc1effd700 1 --2- 172.21.15.88:0/407308505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc200feb50 0x7fbc20069500 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbc1000c970 tx=0x7fbc1000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:01.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.893+0000 7fbc248a9700 1 -- 172.21.15.88:0/407308505 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc10004220 con 0x7fbc200feb50 2023-07-14T17:43:01.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.893+0000 7fbc248a9700 1 -- 172.21.15.88:0/407308505 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fbc10007d30 con 0x7fbc200feb50 2023-07-14T17:43:01.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.893+0000 7fbc248a9700 1 -- 172.21.15.88:0/407308505 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc10003bf0 con 0x7fbc200feb50 2023-07-14T17:43:01.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.893+0000 7fbc258ab700 1 -- 172.21.15.88:0/407308505 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc2019da90 con 0x7fbc200feb50 2023-07-14T17:43:01.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.893+0000 7fbc258ab700 1 -- 172.21.15.88:0/407308505 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2019df50 con 0x7fbc200feb50 2023-07-14T17:43:01.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.894+0000 7fbc258ab700 1 -- 172.21.15.88:0/407308505 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc201060a0 con 0x7fbc200feb50 2023-07-14T17:43:01.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.895+0000 7fbc248a9700 1 -- 172.21.15.88:0/407308505 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbc100078a0 con 0x7fbc200feb50 2023-07-14T17:43:01.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.895+0000 7fbc248a9700 1 --2- 172.21.15.88:0/407308505 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbc0c0a1150 0x7fbc0c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:01.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.895+0000 7fbc1e7fc700 1 --2- 172.21.15.88:0/407308505 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbc0c0a1150 0x7fbc0c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:01.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.895+0000 7fbc248a9700 1 -- 172.21.15.88:0/407308505 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbc100cc780 con 0x7fbc200feb50 2023-07-14T17:43:01.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.896+0000 7fbc1e7fc700 1 --2- 172.21.15.88:0/407308505 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbc0c0a1150 0x7fbc0c0a3600 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbc08005fd0 tx=0x7fbc08005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:01.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:01.899+0000 7fbc248a9700 1 -- 172.21.15.88:0/407308505 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbc10095ce0 con 0x7fbc200feb50 2023-07-14T17:43:02.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.133+0000 7fbc258ab700 1 -- 172.21.15.88:0/407308505 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fbc2002c8e0 con 0x7fbc200feb50 2023-07-14T17:43:02.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:01 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2639073105' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T17:43:02.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:01 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] up:active 2023-07-14T17:43:02.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:01 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2639073105' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T17:43:02.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:01 smithi088 ceph-mon[127465]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T17:43:02.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:01 smithi088 ceph-mon[127465]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2023-07-14T17:43:02.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:01 smithi088 ceph-mon[127465]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2023-07-14T17:43:02.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:01 smithi088 ceph-mon[127465]: daemon mds.i is now active in filesystem cephfs as rank 2 2023-07-14T17:43:02.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:01 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2639073105' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T17:43:02.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:01 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] up:active 2023-07-14T17:43:02.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:01 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2639073105' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T17:43:02.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:01 smithi107 ceph-mon[128999]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T17:43:02.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:01 smithi107 ceph-mon[128999]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2023-07-14T17:43:02.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:01 smithi107 ceph-mon[128999]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2023-07-14T17:43:02.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:01 smithi107 ceph-mon[128999]: daemon mds.i is now active in filesystem cephfs as rank 2 2023-07-14T17:43:02.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:01 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2639073105' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T17:43:02.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:01 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] up:active 2023-07-14T17:43:02.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:01 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2639073105' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T17:43:02.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:01 smithi178 ceph-mon[129368]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T17:43:02.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:01 smithi178 ceph-mon[129368]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2023-07-14T17:43:02.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:01 smithi178 ceph-mon[129368]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2023-07-14T17:43:02.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:01 smithi178 ceph-mon[129368]: daemon mds.i is now active in filesystem cephfs as rank 2 2023-07-14T17:43:02.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.749+0000 7fbc248a9700 1 -- 172.21.15.88:0/407308505 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7fbc10099300 con 0x7fbc200feb50 2023-07-14T17:43:02.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.752+0000 7fbc258ab700 1 -- 172.21.15.88:0/407308505 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbc0c0a1150 msgr2=0x7fbc0c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:02.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.752+0000 7fbc258ab700 1 --2- 172.21.15.88:0/407308505 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbc0c0a1150 0x7fbc0c0a3600 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbc08005fd0 tx=0x7fbc08005e30 comp rx=0 tx=0).stop 2023-07-14T17:43:02.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.752+0000 7fbc258ab700 1 -- 172.21.15.88:0/407308505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc200feb50 msgr2=0x7fbc20069500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:02.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.752+0000 7fbc258ab700 1 --2- 172.21.15.88:0/407308505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc200feb50 0x7fbc20069500 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbc1000c970 tx=0x7fbc1000cd30 comp rx=0 tx=0).stop 2023-07-14T17:43:02.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.752+0000 7fbc258ab700 1 -- 172.21.15.88:0/407308505 shutdown_connections 2023-07-14T17:43:02.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.752+0000 7fbc258ab700 1 --2- 172.21.15.88:0/407308505 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbc201006d0 0x7fbc2019cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:02.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.752+0000 7fbc258ab700 1 --2- 172.21.15.88:0/407308505 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbc0c0a1150 0x7fbc0c0a3600 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:02.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.752+0000 7fbc258ab700 1 --2- 172.21.15.88:0/407308505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc200feb50 0x7fbc20069500 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:02.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.752+0000 7fbc258ab700 1 --2- 172.21.15.88:0/407308505 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbc200ffd40 0x7fbc2019a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:02.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.752+0000 7fbc258ab700 1 -- 172.21.15.88:0/407308505 >> 172.21.15.88:0/407308505 conn(0x7fbc200fa150 msgr2=0x7fbc201031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:02.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.752+0000 7fbc258ab700 1 -- 172.21.15.88:0/407308505 shutdown_connections 2023-07-14T17:43:02.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.752+0000 7fbc258ab700 1 -- 172.21.15.88:0/407308505 wait complete. 2023-07-14T17:43:02.750 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2023-07-14T17:43:02.750 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 '' 2023-07-14T17:43:02.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.894+0000 7f7a60585700 1 -- 172.21.15.88:0/3393603938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a580fe2a0 msgr2=0x7f7a580fe6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:02.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.895+0000 7f7a60585700 1 --2- 172.21.15.88:0/3393603938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a580fe2a0 0x7f7a580fe6b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7a54009990 tx=0x7f7a54009ca0 comp rx=0 tx=0).stop 2023-07-14T17:43:02.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.895+0000 7f7a60585700 1 -- 172.21.15.88:0/3393603938 shutdown_connections 2023-07-14T17:43:02.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.895+0000 7f7a60585700 1 --2- 172.21.15.88:0/3393603938 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a580ffee0 0x7f7a58106780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:02.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.895+0000 7f7a60585700 1 --2- 172.21.15.88:0/3393603938 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a580fe2a0 0x7f7a580fe6b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:02.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.895+0000 7f7a60585700 1 --2- 172.21.15.88:0/3393603938 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a580ff530 0x7f7a580ff9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:02.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.895+0000 7f7a60585700 1 -- 172.21.15.88:0/3393603938 >> 172.21.15.88:0/3393603938 conn(0x7f7a580f98a0 msgr2=0x7f7a580fbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:02.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.895+0000 7f7a60585700 1 -- 172.21.15.88:0/3393603938 shutdown_connections 2023-07-14T17:43:02.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.895+0000 7f7a60585700 1 -- 172.21.15.88:0/3393603938 wait complete. 2023-07-14T17:43:02.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.895+0000 7f7a60585700 1 Processor -- start 2023-07-14T17:43:02.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.895+0000 7f7a60585700 1 -- start start 2023-07-14T17:43:02.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.895+0000 7f7a60585700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a580fe2a0 0x7f7a58194740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:02.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.895+0000 7f7a60585700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a580ff530 0x7f7a58194c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:02.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.895+0000 7f7a60585700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a580ffee0 0x7f7a5819bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:02.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.895+0000 7f7a60585700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5819c2c0 con 0x7f7a580ffee0 2023-07-14T17:43:02.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.895+0000 7f7a60585700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5819c400 con 0x7f7a580ff530 2023-07-14T17:43:02.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.895+0000 7f7a60585700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5819c590 con 0x7f7a580fe2a0 2023-07-14T17:43:02.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.896+0000 7f7a5eb22700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a580ffee0 0x7f7a5819bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:02.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.896+0000 7f7a5eb22700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a580ffee0 0x7f7a5819bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41900/0 (socket says 172.21.15.88:41900) 2023-07-14T17:43:02.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.896+0000 7f7a5eb22700 1 -- 172.21.15.88:0/2555393314 learned_addr learned my addr 172.21.15.88:0/2555393314 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:02.885 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.896+0000 7f7a5db20700 1 --2- 172.21.15.88:0/2555393314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a580ff530 0x7f7a58194c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:02.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.896+0000 7f7a5e321700 1 --2- 172.21.15.88:0/2555393314 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a580fe2a0 0x7f7a58194740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:02.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.896+0000 7f7a5eb22700 1 -- 172.21.15.88:0/2555393314 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a580fe2a0 msgr2=0x7f7a58194740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:02.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.896+0000 7f7a5eb22700 1 --2- 172.21.15.88:0/2555393314 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a580fe2a0 0x7f7a58194740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:02.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.896+0000 7f7a5eb22700 1 -- 172.21.15.88:0/2555393314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a580ff530 msgr2=0x7f7a58194c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:02.886 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.896+0000 7f7a5eb22700 1 --2- 172.21.15.88:0/2555393314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a580ff530 0x7f7a58194c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:02.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.896+0000 7f7a5eb22700 1 -- 172.21.15.88:0/2555393314 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a540095f0 con 0x7f7a580ffee0 2023-07-14T17:43:02.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.896+0000 7f7a5e321700 1 --2- 172.21.15.88:0/2555393314 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a580fe2a0 0x7f7a58194740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:43:02.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.896+0000 7f7a5eb22700 1 --2- 172.21.15.88:0/2555393314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a580ffee0 0x7f7a5819bcf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7a50007d70 tx=0x7f7a5000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:02.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.896+0000 7f7a4f7fe700 1 -- 172.21.15.88:0/2555393314 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5000ad40 con 0x7f7a580ffee0 2023-07-14T17:43:02.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.896+0000 7f7a4f7fe700 1 -- 172.21.15.88:0/2555393314 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7a500128b0 con 0x7f7a580ffee0 2023-07-14T17:43:02.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.896+0000 7f7a60585700 1 -- 172.21.15.88:0/2555393314 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a5819c870 con 0x7f7a580ffee0 2023-07-14T17:43:02.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.896+0000 7f7a4f7fe700 1 -- 172.21.15.88:0/2555393314 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a500115a0 con 0x7f7a580ffee0 2023-07-14T17:43:02.887 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.896+0000 7f7a60585700 1 -- 172.21.15.88:0/2555393314 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a5819cd30 con 0x7f7a580ffee0 2023-07-14T17:43:02.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.897+0000 7f7a60585700 1 -- 172.21.15.88:0/2555393314 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a5810a1b0 con 0x7f7a580ffee0 2023-07-14T17:43:02.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.898+0000 7f7a4f7fe700 1 -- 172.21.15.88:0/2555393314 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7a50012a20 con 0x7f7a580ffee0 2023-07-14T17:43:02.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.899+0000 7f7a4f7fe700 1 --2- 172.21.15.88:0/2555393314 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7a440a1150 0x7f7a440a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:02.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.899+0000 7f7a5e321700 1 --2- 172.21.15.88:0/2555393314 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7a440a1150 0x7f7a440a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:02.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.899+0000 7f7a4f7fe700 1 -- 172.21.15.88:0/2555393314 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7a500ccd70 con 0x7f7a580ffee0 2023-07-14T17:43:02.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.899+0000 7f7a5e321700 1 --2- 172.21.15.88:0/2555393314 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7a440a1150 0x7f7a440a3600 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7a5400b5c0 tx=0x7f7a54005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:02.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:02.903+0000 7f7a4f7fe700 1 -- 172.21.15.88:0/2555393314 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7a500962d0 con 0x7f7a580ffee0 2023-07-14T17:43:03.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.123+0000 7f7a60585700 1 -- 172.21.15.88:0/2555393314 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a58106e30 con 0x7f7a440a1150 2023-07-14T17:43:03.112 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:02 smithi088 ceph-mon[127465]: pgmap v265: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-14T17:43:03.112 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:02 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/407308505' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T17:43:03.113 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:02 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] up:active 2023-07-14T17:43:03.113 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:02 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/407308505' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T17:43:03.113 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:02 smithi088 ceph-mon[127465]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-14T17:43:03.114 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:02 smithi088 ceph-mon[127465]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2023-07-14T17:43:03.114 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:02 smithi088 ceph-mon[127465]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2023-07-14T17:43:03.114 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:02 smithi088 ceph-mon[127465]: daemon mds.c is now active in filesystem cephfs as rank 3 2023-07-14T17:43:03.224 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:43:02 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-b[154079]: 2023-07-14T17:43:02.890+0000 7fb592e7d700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-07-14T17:43:03.225 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:43:02 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-b[154079]: 2023-07-14T17:43:02.890+0000 7fb592e7d700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-07-14T17:43:03.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:02 smithi107 ceph-mon[128999]: pgmap v265: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-14T17:43:03.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:02 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/407308505' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T17:43:03.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:02 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] up:active 2023-07-14T17:43:03.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:02 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/407308505' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T17:43:03.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:02 smithi107 ceph-mon[128999]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-14T17:43:03.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:02 smithi107 ceph-mon[128999]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2023-07-14T17:43:03.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:02 smithi107 ceph-mon[128999]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2023-07-14T17:43:03.227 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:02 smithi107 ceph-mon[128999]: daemon mds.c is now active in filesystem cephfs as rank 3 2023-07-14T17:43:03.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:02 smithi178 ceph-mon[129368]: pgmap v265: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-14T17:43:03.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:02 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/407308505' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T17:43:03.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:02 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] up:active 2023-07-14T17:43:03.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:02 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/407308505' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T17:43:03.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:02 smithi178 ceph-mon[129368]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-14T17:43:03.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:02 smithi178 ceph-mon[129368]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2023-07-14T17:43:03.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:02 smithi178 ceph-mon[129368]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2023-07-14T17:43:03.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:02 smithi178 ceph-mon[129368]: daemon mds.c is now active in filesystem cephfs as rank 3 2023-07-14T17:43:03.442 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:43:03 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:43:03.128+0000 7f4490462700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T17:43:03.442 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:43:03 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:43:03.128+0000 7f4490462700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T17:43:03.442 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:43:03 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:43:03.128+0000 7f4490462700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T17:43:03.442 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:43:03 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:43:03.128+0000 7f4490462700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T17:43:03.442 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:43:03 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:43:03.128+0000 7f4490462700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T17:43:03.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.794+0000 7f7a4f7fe700 1 -- 172.21.15.88:0/2555393314 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7a58106e30 con 0x7f7a440a1150 2023-07-14T17:43:03.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.796+0000 7f7a60585700 1 -- 172.21.15.88:0/2555393314 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7a440a1150 msgr2=0x7f7a440a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:03.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.796+0000 7f7a60585700 1 --2- 172.21.15.88:0/2555393314 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7a440a1150 0x7f7a440a3600 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7a5400b5c0 tx=0x7f7a54005fb0 comp rx=0 tx=0).stop 2023-07-14T17:43:03.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.796+0000 7f7a60585700 1 -- 172.21.15.88:0/2555393314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a580ffee0 msgr2=0x7f7a5819bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:03.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.796+0000 7f7a60585700 1 --2- 172.21.15.88:0/2555393314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a580ffee0 0x7f7a5819bcf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7a50007d70 tx=0x7f7a5000a340 comp rx=0 tx=0).stop 2023-07-14T17:43:03.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.796+0000 7f7a60585700 1 -- 172.21.15.88:0/2555393314 shutdown_connections 2023-07-14T17:43:03.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.796+0000 7f7a60585700 1 --2- 172.21.15.88:0/2555393314 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7a440a1150 0x7f7a440a3600 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:03.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.796+0000 7f7a60585700 1 --2- 172.21.15.88:0/2555393314 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a580ffee0 0x7f7a5819bcf0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:03.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.796+0000 7f7a60585700 1 --2- 172.21.15.88:0/2555393314 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7a580fe2a0 0x7f7a58194740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:03.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.796+0000 7f7a60585700 1 --2- 172.21.15.88:0/2555393314 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7a580ff530 0x7f7a58194c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:03.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.796+0000 7f7a60585700 1 -- 172.21.15.88:0/2555393314 >> 172.21.15.88:0/2555393314 conn(0x7f7a580f98a0 msgr2=0x7f7a58102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:03.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.796+0000 7f7a60585700 1 -- 172.21.15.88:0/2555393314 shutdown_connections 2023-07-14T17:43:03.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.796+0000 7f7a60585700 1 -- 172.21.15.88:0/2555393314 wait complete. 2023-07-14T17:43:03.791 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_0 2023-07-14T17:43:03.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.936+0000 7f15f0291700 1 -- 172.21.15.88:0/2886011187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15e80fed60 msgr2=0x7f15e80ff1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:03.918 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.936+0000 7f15f0291700 1 --2- 172.21.15.88:0/2886011187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15e80fed60 0x7f15e80ff1d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f15e4009990 tx=0x7f15e4009ca0 comp rx=0 tx=0).stop 2023-07-14T17:43:03.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.936+0000 7f15f0291700 1 -- 172.21.15.88:0/2886011187 shutdown_connections 2023-07-14T17:43:03.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.936+0000 7f15f0291700 1 --2- 172.21.15.88:0/2886011187 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15e80ff710 0x7f15e8106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:03.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.936+0000 7f15f0291700 1 --2- 172.21.15.88:0/2886011187 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15e8100110 0x7f15e8100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:03.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.936+0000 7f15f0291700 1 --2- 172.21.15.88:0/2886011187 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15e80fed60 0x7f15e80ff1d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:03.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.936+0000 7f15f0291700 1 -- 172.21.15.88:0/2886011187 >> 172.21.15.88:0/2886011187 conn(0x7f15e80f98a0 msgr2=0x7f15e80fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:03.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.936+0000 7f15f0291700 1 -- 172.21.15.88:0/2886011187 shutdown_connections 2023-07-14T17:43:03.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.936+0000 7f15f0291700 1 -- 172.21.15.88:0/2886011187 wait complete. 2023-07-14T17:43:03.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.936+0000 7f15f0291700 1 Processor -- start 2023-07-14T17:43:03.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.936+0000 7f15f0291700 1 -- start start 2023-07-14T17:43:03.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.936+0000 7f15f0291700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15e80fed60 0x7f15e8102b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:03.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.936+0000 7f15f0291700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15e80ff710 0x7f15e8101150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:03.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.936+0000 7f15f0291700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15e8100110 0x7f15e8101690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:03.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.936+0000 7f15f0291700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e8101e30 con 0x7f15e80ff710 2023-07-14T17:43:03.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.936+0000 7f15f0291700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e8101fa0 con 0x7f15e8100110 2023-07-14T17:43:03.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.936+0000 7f15f0291700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e819f930 con 0x7f15e80fed60 2023-07-14T17:43:03.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.937+0000 7f15ed82c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15e80ff710 0x7f15e8101150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:03.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.937+0000 7f15ed82c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15e80ff710 0x7f15e8101150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41932/0 (socket says 172.21.15.88:41932) 2023-07-14T17:43:03.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.937+0000 7f15ed82c700 1 -- 172.21.15.88:0/2648128192 learned_addr learned my addr 172.21.15.88:0/2648128192 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:03.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.937+0000 7f15ee02d700 1 --2- 172.21.15.88:0/2648128192 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15e80fed60 0x7f15e8102b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:03.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.937+0000 7f15ee82e700 1 --2- 172.21.15.88:0/2648128192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15e8100110 0x7f15e8101690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:03.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.937+0000 7f15ed82c700 1 -- 172.21.15.88:0/2648128192 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15e80fed60 msgr2=0x7f15e8102b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:03.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.937+0000 7f15ed82c700 1 --2- 172.21.15.88:0/2648128192 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15e80fed60 0x7f15e8102b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:03.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.937+0000 7f15ed82c700 1 -- 172.21.15.88:0/2648128192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15e8100110 msgr2=0x7f15e8101690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:03.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.937+0000 7f15ed82c700 1 --2- 172.21.15.88:0/2648128192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15e8100110 0x7f15e8101690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:03.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.937+0000 7f15ed82c700 1 -- 172.21.15.88:0/2648128192 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e40095f0 con 0x7f15e80ff710 2023-07-14T17:43:03.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.937+0000 7f15ee02d700 1 --2- 172.21.15.88:0/2648128192 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15e80fed60 0x7f15e8102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:43:03.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.937+0000 7f15ee82e700 1 --2- 172.21.15.88:0/2648128192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15e8100110 0x7f15e8101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:43:03.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.937+0000 7f15ed82c700 1 --2- 172.21.15.88:0/2648128192 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15e80ff710 0x7f15e8101150 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f15e4006010 tx=0x7f15e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:03.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.937+0000 7f15df7fe700 1 -- 172.21.15.88:0/2648128192 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e401d070 con 0x7f15e80ff710 2023-07-14T17:43:03.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.937+0000 7f15df7fe700 1 -- 172.21.15.88:0/2648128192 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f15e40041b0 con 0x7f15e80ff710 2023-07-14T17:43:03.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.937+0000 7f15f0291700 1 -- 172.21.15.88:0/2648128192 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e819fbb0 con 0x7f15e80ff710 2023-07-14T17:43:03.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.937+0000 7f15df7fe700 1 -- 172.21.15.88:0/2648128192 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e4022da0 con 0x7f15e80ff710 2023-07-14T17:43:03.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.937+0000 7f15f0291700 1 -- 172.21.15.88:0/2648128192 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e819ffd0 con 0x7f15e80ff710 2023-07-14T17:43:03.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.938+0000 7f15f0291700 1 -- 172.21.15.88:0/2648128192 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e810a200 con 0x7f15e80ff710 2023-07-14T17:43:03.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.939+0000 7f15df7fe700 1 -- 172.21.15.88:0/2648128192 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f15e4022570 con 0x7f15e80ff710 2023-07-14T17:43:03.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.940+0000 7f15df7fe700 1 --2- 172.21.15.88:0/2648128192 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f15d40a1150 0x7f15d40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:03.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.940+0000 7f15df7fe700 1 -- 172.21.15.88:0/2648128192 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f15e40ce440 con 0x7f15e80ff710 2023-07-14T17:43:03.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.940+0000 7f15ee02d700 1 --2- 172.21.15.88:0/2648128192 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f15d40a1150 0x7f15d40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:03.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.940+0000 7f15ee02d700 1 --2- 172.21.15.88:0/2648128192 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f15d40a1150 0x7f15d40a3600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f15d8005950 tx=0x7f15d80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:03.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:03.943+0000 7f15df7fe700 1 -- 172.21.15.88:0/2648128192 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f15e4097a30 con 0x7f15e80ff710 2023-07-14T17:43:04.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.165+0000 7f15f0291700 1 -- 172.21.15.88:0/2648128192 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f15e80fba10 con 0x7f15d40a1150 2023-07-14T17:43:04.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.169+0000 7f15df7fe700 1 -- 172.21.15.88:0/2648128192 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f15e80fba10 con 0x7f15d40a1150 2023-07-14T17:43:04.151 INFO:teuthology.orchestra.run.smithi088.stdout:/volumes/_nogroup/sv_0/9513faeb-71d8-4552-bbbd-ba63e6a250d0 2023-07-14T17:43:04.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.171+0000 7f15f0291700 1 -- 172.21.15.88:0/2648128192 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f15d40a1150 msgr2=0x7f15d40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:04.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.171+0000 7f15f0291700 1 --2- 172.21.15.88:0/2648128192 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f15d40a1150 0x7f15d40a3600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f15d8005950 tx=0x7f15d80058e0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.171+0000 7f15f0291700 1 -- 172.21.15.88:0/2648128192 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15e80ff710 msgr2=0x7f15e8101150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:04.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.171+0000 7f15f0291700 1 --2- 172.21.15.88:0/2648128192 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15e80ff710 0x7f15e8101150 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f15e4006010 tx=0x7f15e4004730 comp rx=0 tx=0).stop 2023-07-14T17:43:04.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.171+0000 7f15f0291700 1 -- 172.21.15.88:0/2648128192 shutdown_connections 2023-07-14T17:43:04.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.171+0000 7f15f0291700 1 --2- 172.21.15.88:0/2648128192 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f15e8100110 0x7f15e8101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.171+0000 7f15f0291700 1 --2- 172.21.15.88:0/2648128192 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f15e80fed60 0x7f15e8102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.171+0000 7f15f0291700 1 --2- 172.21.15.88:0/2648128192 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f15d40a1150 0x7f15d40a3600 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.171+0000 7f15f0291700 1 --2- 172.21.15.88:0/2648128192 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15e80ff710 0x7f15e8101150 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.171+0000 7f15f0291700 1 -- 172.21.15.88:0/2648128192 >> 172.21.15.88:0/2648128192 conn(0x7f15e80f98a0 msgr2=0x7f15e80fb300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:04.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.171+0000 7f15f0291700 1 -- 172.21.15.88:0/2648128192 shutdown_connections 2023-07-14T17:43:04.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.171+0000 7f15f0291700 1 -- 172.21.15.88:0/2648128192 wait complete. 2023-07-14T17:43:04.164 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:03 smithi088 ceph-mon[127465]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:43:04.164 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:03 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] up:active 2023-07-14T17:43:04.164 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:03 smithi088 ceph-mon[127465]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-14T17:43:04.165 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:03 smithi088 ceph-mon[127465]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T17:43:04.165 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:03 smithi088 ceph-mon[127465]: Cluster is now healthy 2023-07-14T17:43:04.165 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:03 smithi088 ceph-mon[127465]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2023-07-14T17:43:04.165 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:03 smithi088 ceph-mon[127465]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2023-07-14T17:43:04.165 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:03 smithi088 ceph-mon[127465]: daemon mds.e is now active in filesystem cephfs as rank 4 2023-07-14T17:43:04.166 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 '' 2023-07-14T17:43:04.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:03 smithi107 ceph-mon[128999]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:43:04.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:03 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] up:active 2023-07-14T17:43:04.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:03 smithi107 ceph-mon[128999]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-14T17:43:04.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:03 smithi107 ceph-mon[128999]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T17:43:04.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:03 smithi107 ceph-mon[128999]: Cluster is now healthy 2023-07-14T17:43:04.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:03 smithi107 ceph-mon[128999]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2023-07-14T17:43:04.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:03 smithi107 ceph-mon[128999]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2023-07-14T17:43:04.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:03 smithi107 ceph-mon[128999]: daemon mds.e is now active in filesystem cephfs as rank 4 2023-07-14T17:43:04.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.309+0000 7f9e78f48700 1 -- 172.21.15.88:0/1875297270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e74100690 msgr2=0x7f9e74106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:04.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.309+0000 7f9e78f48700 1 --2- 172.21.15.88:0/1875297270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e74100690 0x7f9e74106f50 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c00b0a0 tx=0x7f9e6c00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.309+0000 7f9e78f48700 1 -- 172.21.15.88:0/1875297270 shutdown_connections 2023-07-14T17:43:04.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.309+0000 7f9e78f48700 1 --2- 172.21.15.88:0/1875297270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e74100690 0x7f9e74106f50 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.309+0000 7f9e78f48700 1 --2- 172.21.15.88:0/1875297270 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e740feb10 0x7f9e740fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.309+0000 7f9e78f48700 1 --2- 172.21.15.88:0/1875297270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e740ffd00 0x7f9e74100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.292 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.309+0000 7f9e78f48700 1 -- 172.21.15.88:0/1875297270 >> 172.21.15.88:0/1875297270 conn(0x7f9e740fa130 msgr2=0x7f9e740fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:04.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.309+0000 7f9e78f48700 1 -- 172.21.15.88:0/1875297270 shutdown_connections 2023-07-14T17:43:04.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.309+0000 7f9e78f48700 1 -- 172.21.15.88:0/1875297270 wait complete. 2023-07-14T17:43:04.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.310+0000 7f9e78f48700 1 Processor -- start 2023-07-14T17:43:04.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.310+0000 7f9e78f48700 1 -- start start 2023-07-14T17:43:04.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.310+0000 7f9e78f48700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e740feb10 0x7f9e741948c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:04.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.310+0000 7f9e78f48700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e740ffd00 0x7f9e74194e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:04.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.310+0000 7f9e78f48700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e74100690 0x7f9e7419be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:04.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.310+0000 7f9e78f48700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7419c440 con 0x7f9e740ffd00 2023-07-14T17:43:04.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.310+0000 7f9e78f48700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7419c580 con 0x7f9e74100690 2023-07-14T17:43:04.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.310+0000 7f9e78f48700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7419c710 con 0x7f9e740feb10 2023-07-14T17:43:04.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.310+0000 7f9e71d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e740ffd00 0x7f9e74194e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:04.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.310+0000 7f9e71d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e740ffd00 0x7f9e74194e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41944/0 (socket says 172.21.15.88:41944) 2023-07-14T17:43:04.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.310+0000 7f9e71d9b700 1 -- 172.21.15.88:0/2719103199 learned_addr learned my addr 172.21.15.88:0/2719103199 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:04.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.311+0000 7f9e7259c700 1 --2- 172.21.15.88:0/2719103199 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e740feb10 0x7f9e741948c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:04.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.311+0000 7f9e72d9d700 1 --2- 172.21.15.88:0/2719103199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e74100690 0x7f9e7419be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:04.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.311+0000 7f9e71d9b700 1 -- 172.21.15.88:0/2719103199 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e740feb10 msgr2=0x7f9e741948c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:04.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.311+0000 7f9e71d9b700 1 --2- 172.21.15.88:0/2719103199 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e740feb10 0x7f9e741948c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.311+0000 7f9e71d9b700 1 -- 172.21.15.88:0/2719103199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e74100690 msgr2=0x7f9e7419be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:04.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.311+0000 7f9e71d9b700 1 --2- 172.21.15.88:0/2719103199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e74100690 0x7f9e7419be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.311+0000 7f9e71d9b700 1 -- 172.21.15.88:0/2719103199 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6c009d00 con 0x7f9e740ffd00 2023-07-14T17:43:04.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.311+0000 7f9e7259c700 1 --2- 172.21.15.88:0/2719103199 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e740feb10 0x7f9e741948c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:43:04.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.311+0000 7f9e72d9d700 1 --2- 172.21.15.88:0/2719103199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e74100690 0x7f9e7419be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:43:04.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.311+0000 7f9e71d9b700 1 --2- 172.21.15.88:0/2719103199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e740ffd00 0x7f9e74194e00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c00d950 tx=0x7f9e5c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:04.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.311+0000 7f9e6b7fe700 1 -- 172.21.15.88:0/2719103199 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c009b40 con 0x7f9e740ffd00 2023-07-14T17:43:04.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.311+0000 7f9e78f48700 1 -- 172.21.15.88:0/2719103199 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7419c9f0 con 0x7f9e740ffd00 2023-07-14T17:43:04.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.311+0000 7f9e6b7fe700 1 -- 172.21.15.88:0/2719103199 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9e5c009ca0 con 0x7f9e740ffd00 2023-07-14T17:43:04.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.311+0000 7f9e6b7fe700 1 -- 172.21.15.88:0/2719103199 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c00f620 con 0x7f9e740ffd00 2023-07-14T17:43:04.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.311+0000 7f9e78f48700 1 -- 172.21.15.88:0/2719103199 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7419cf50 con 0x7f9e740ffd00 2023-07-14T17:43:04.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.312+0000 7f9e78f48700 1 -- 172.21.15.88:0/2719103199 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e741060a0 con 0x7f9e740ffd00 2023-07-14T17:43:04.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.313+0000 7f9e6b7fe700 1 -- 172.21.15.88:0/2719103199 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9e5c010460 con 0x7f9e740ffd00 2023-07-14T17:43:04.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.313+0000 7f9e6b7fe700 1 --2- 172.21.15.88:0/2719103199 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9e600a1150 0x7f9e600a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:04.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.314+0000 7f9e7259c700 1 --2- 172.21.15.88:0/2719103199 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9e600a1150 0x7f9e600a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:04.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.314+0000 7f9e6b7fe700 1 -- 172.21.15.88:0/2719103199 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9e5c0cce00 con 0x7f9e740ffd00 2023-07-14T17:43:04.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.314+0000 7f9e7259c700 1 --2- 172.21.15.88:0/2719103199 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9e600a1150 0x7f9e600a3600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9e64005950 tx=0x7f9e640058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:04.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.318+0000 7f9e6b7fe700 1 -- 172.21.15.88:0/2719103199 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9e5c096360 con 0x7f9e740ffd00 2023-07-14T17:43:04.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:03 smithi178 ceph-mon[129368]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:43:04.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:03 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] up:active 2023-07-14T17:43:04.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:03 smithi178 ceph-mon[129368]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-14T17:43:04.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:03 smithi178 ceph-mon[129368]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T17:43:04.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:03 smithi178 ceph-mon[129368]: Cluster is now healthy 2023-07-14T17:43:04.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:03 smithi178 ceph-mon[129368]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2023-07-14T17:43:04.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:03 smithi178 ceph-mon[129368]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2023-07-14T17:43:04.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:03 smithi178 ceph-mon[129368]: daemon mds.e is now active in filesystem cephfs as rank 4 2023-07-14T17:43:04.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.538+0000 7f9e78f48700 1 -- 172.21.15.88:0/2719103199 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e74060a80 con 0x7f9e600a1150 2023-07-14T17:43:04.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.563+0000 7f9e6b7fe700 1 -- 172.21.15.88:0/2719103199 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9e74060a80 con 0x7f9e600a1150 2023-07-14T17:43:04.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.565+0000 7f9e78f48700 1 -- 172.21.15.88:0/2719103199 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9e600a1150 msgr2=0x7f9e600a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:04.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.565+0000 7f9e78f48700 1 --2- 172.21.15.88:0/2719103199 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9e600a1150 0x7f9e600a3600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9e64005950 tx=0x7f9e640058e0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.565+0000 7f9e78f48700 1 -- 172.21.15.88:0/2719103199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e740ffd00 msgr2=0x7f9e74194e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:04.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.565+0000 7f9e78f48700 1 --2- 172.21.15.88:0/2719103199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e740ffd00 0x7f9e74194e00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c00d950 tx=0x7f9e5c00dd10 comp rx=0 tx=0).stop 2023-07-14T17:43:04.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.565+0000 7f9e78f48700 1 -- 172.21.15.88:0/2719103199 shutdown_connections 2023-07-14T17:43:04.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.565+0000 7f9e78f48700 1 --2- 172.21.15.88:0/2719103199 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9e74100690 0x7f9e7419be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.565+0000 7f9e78f48700 1 --2- 172.21.15.88:0/2719103199 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9e740feb10 0x7f9e741948c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.565+0000 7f9e78f48700 1 --2- 172.21.15.88:0/2719103199 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9e600a1150 0x7f9e600a3600 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.565+0000 7f9e78f48700 1 --2- 172.21.15.88:0/2719103199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9e740ffd00 0x7f9e74194e00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.565+0000 7f9e78f48700 1 -- 172.21.15.88:0/2719103199 >> 172.21.15.88:0/2719103199 conn(0x7f9e740fa130 msgr2=0x7f9e741031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:04.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.566+0000 7f9e78f48700 1 -- 172.21.15.88:0/2719103199 shutdown_connections 2023-07-14T17:43:04.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.566+0000 7f9e78f48700 1 -- 172.21.15.88:0/2719103199 wait complete. 2023-07-14T17:43:04.560 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_1 2023-07-14T17:43:04.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.702+0000 7fc948a19700 1 -- 172.21.15.88:0/1865389601 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9440ffd00 msgr2=0x7fc944100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:04.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.702+0000 7fc948a19700 1 --2- 172.21.15.88:0/1865389601 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9440ffd00 0x7fc944100150 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc934009990 tx=0x7fc934009ca0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.702+0000 7fc948a19700 1 -- 172.21.15.88:0/1865389601 shutdown_connections 2023-07-14T17:43:04.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.702+0000 7fc948a19700 1 --2- 172.21.15.88:0/1865389601 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc944100690 0x7fc944106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.702+0000 7fc948a19700 1 --2- 172.21.15.88:0/1865389601 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9440feb10 0x7fc9440fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.702+0000 7fc948a19700 1 --2- 172.21.15.88:0/1865389601 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9440ffd00 0x7fc944100150 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.702+0000 7fc948a19700 1 -- 172.21.15.88:0/1865389601 >> 172.21.15.88:0/1865389601 conn(0x7fc9440fa130 msgr2=0x7fc9440fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:04.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.702+0000 7fc948a19700 1 -- 172.21.15.88:0/1865389601 shutdown_connections 2023-07-14T17:43:04.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.702+0000 7fc948a19700 1 -- 172.21.15.88:0/1865389601 wait complete. 2023-07-14T17:43:04.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc948a19700 1 Processor -- start 2023-07-14T17:43:04.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc948a19700 1 -- start start 2023-07-14T17:43:04.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc948a19700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9440feb10 0x7fc9441957d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:04.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc948a19700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9440ffd00 0x7fc944195d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:04.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc948a19700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc944100690 0x7fc94419cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:04.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc948a19700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94419d350 con 0x7fc9440ffd00 2023-07-14T17:43:04.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc948a19700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94419d490 con 0x7fc9440feb10 2023-07-14T17:43:04.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc948a19700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94419d620 con 0x7fc944100690 2023-07-14T17:43:04.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc941d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9440ffd00 0x7fc944195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:04.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc941d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9440ffd00 0x7fc944195d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41962/0 (socket says 172.21.15.88:41962) 2023-07-14T17:43:04.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc941d9b700 1 -- 172.21.15.88:0/4185955751 learned_addr learned my addr 172.21.15.88:0/4185955751 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:04.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc94259c700 1 --2- 172.21.15.88:0/4185955751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9440feb10 0x7fc9441957d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:04.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc942d9d700 1 --2- 172.21.15.88:0/4185955751 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc944100690 0x7fc94419cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:04.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc941d9b700 1 -- 172.21.15.88:0/4185955751 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc944100690 msgr2=0x7fc94419cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:04.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc941d9b700 1 --2- 172.21.15.88:0/4185955751 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc944100690 0x7fc94419cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc941d9b700 1 -- 172.21.15.88:0/4185955751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9440feb10 msgr2=0x7fc9441957d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:04.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc941d9b700 1 --2- 172.21.15.88:0/4185955751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9440feb10 0x7fc9441957d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc941d9b700 1 -- 172.21.15.88:0/4185955751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9340095f0 con 0x7fc9440ffd00 2023-07-14T17:43:04.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.703+0000 7fc942d9d700 1 --2- 172.21.15.88:0/4185955751 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc944100690 0x7fc94419cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:43:04.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.704+0000 7fc941d9b700 1 --2- 172.21.15.88:0/4185955751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9440ffd00 0x7fc944195d10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc934005f50 tx=0x7fc934005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:04.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.704+0000 7fc93b7fe700 1 -- 172.21.15.88:0/4185955751 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93401d070 con 0x7fc9440ffd00 2023-07-14T17:43:04.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.704+0000 7fc93b7fe700 1 -- 172.21.15.88:0/4185955751 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc934004100 con 0x7fc9440ffd00 2023-07-14T17:43:04.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.704+0000 7fc948a19700 1 -- 172.21.15.88:0/4185955751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc94419d8a0 con 0x7fc9440ffd00 2023-07-14T17:43:04.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.704+0000 7fc93b7fe700 1 -- 172.21.15.88:0/4185955751 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc934022dd0 con 0x7fc9440ffd00 2023-07-14T17:43:04.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.704+0000 7fc948a19700 1 -- 172.21.15.88:0/4185955751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94419dcc0 con 0x7fc9440ffd00 2023-07-14T17:43:04.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.705+0000 7fc948a19700 1 -- 172.21.15.88:0/4185955751 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc94410a190 con 0x7fc9440ffd00 2023-07-14T17:43:04.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.706+0000 7fc93b7fe700 1 -- 172.21.15.88:0/4185955751 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc934022570 con 0x7fc9440ffd00 2023-07-14T17:43:04.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.706+0000 7fc93b7fe700 1 --2- 172.21.15.88:0/4185955751 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc9300a1150 0x7fc9300a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:04.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.707+0000 7fc94259c700 1 --2- 172.21.15.88:0/4185955751 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc9300a1150 0x7fc9300a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:04.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.707+0000 7fc93b7fe700 1 -- 172.21.15.88:0/4185955751 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc9340ce0a0 con 0x7fc9440ffd00 2023-07-14T17:43:04.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.707+0000 7fc94259c700 1 --2- 172.21.15.88:0/4185955751 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc9300a1150 0x7fc9300a3600 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc9440ffb60 tx=0x7fc92c00a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:04.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.711+0000 7fc93b7fe700 1 -- 172.21.15.88:0/4185955751 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc934097690 con 0x7fc9440ffd00 2023-07-14T17:43:04.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.773+0000 7fc93b7fe700 1 -- 172.21.15.88:0/4185955751 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc934097a60 con 0x7fc9440ffd00 2023-07-14T17:43:04.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.945+0000 7fc948a19700 1 -- 172.21.15.88:0/4185955751 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc944067ed0 con 0x7fc9300a1150 2023-07-14T17:43:04.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.949+0000 7fc93b7fe700 1 -- 172.21.15.88:0/4185955751 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7fc944067ed0 con 0x7fc9300a1150 2023-07-14T17:43:04.931 INFO:teuthology.orchestra.run.smithi088.stdout:/volumes/_nogroup/sv_1/1870349f-5f79-4b6d-b7dc-d09a59bb36e9 2023-07-14T17:43:04.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.951+0000 7fc948a19700 1 -- 172.21.15.88:0/4185955751 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc9300a1150 msgr2=0x7fc9300a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:04.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.951+0000 7fc948a19700 1 --2- 172.21.15.88:0/4185955751 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc9300a1150 0x7fc9300a3600 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc9440ffb60 tx=0x7fc92c00a3b0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.951+0000 7fc948a19700 1 -- 172.21.15.88:0/4185955751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9440ffd00 msgr2=0x7fc944195d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:04.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.951+0000 7fc948a19700 1 --2- 172.21.15.88:0/4185955751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9440ffd00 0x7fc944195d10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc934005f50 tx=0x7fc934005740 comp rx=0 tx=0).stop 2023-07-14T17:43:04.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.951+0000 7fc948a19700 1 -- 172.21.15.88:0/4185955751 shutdown_connections 2023-07-14T17:43:04.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.951+0000 7fc948a19700 1 --2- 172.21.15.88:0/4185955751 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc944100690 0x7fc94419cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.951+0000 7fc948a19700 1 --2- 172.21.15.88:0/4185955751 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc9440feb10 0x7fc9441957d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.951+0000 7fc948a19700 1 --2- 172.21.15.88:0/4185955751 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc9300a1150 0x7fc9300a3600 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.951+0000 7fc948a19700 1 --2- 172.21.15.88:0/4185955751 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc9440ffd00 0x7fc944195d10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:04.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.951+0000 7fc948a19700 1 -- 172.21.15.88:0/4185955751 >> 172.21.15.88:0/4185955751 conn(0x7fc9440fa130 msgr2=0x7fc9440677c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:04.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.951+0000 7fc948a19700 1 -- 172.21.15.88:0/4185955751 shutdown_connections 2023-07-14T17:43:04.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:04.951+0000 7fc948a19700 1 -- 172.21.15.88:0/4185955751 wait complete. 2023-07-14T17:43:04.946 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:43:05.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.085+0000 7f4b8f8ac700 1 -- 172.21.15.88:0/3326128104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b880ffd40 msgr2=0x7f4b88100190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:05.070 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.085+0000 7f4b8f8ac700 1 --2- 172.21.15.88:0/3326128104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b880ffd40 0x7f4b88100190 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4b84009990 tx=0x7f4b84009ca0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.085+0000 7f4b8f8ac700 1 -- 172.21.15.88:0/3326128104 shutdown_connections 2023-07-14T17:43:05.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.085+0000 7f4b8f8ac700 1 --2- 172.21.15.88:0/3326128104 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b881006d0 0x7f4b88106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.071 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.085+0000 7f4b8f8ac700 1 --2- 172.21.15.88:0/3326128104 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b880feb50 0x7f4b880fef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.085+0000 7f4b8f8ac700 1 --2- 172.21.15.88:0/3326128104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b880ffd40 0x7f4b88100190 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.085+0000 7f4b8f8ac700 1 -- 172.21.15.88:0/3326128104 >> 172.21.15.88:0/3326128104 conn(0x7f4b880fa150 msgr2=0x7f4b880fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:05.072 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.085+0000 7f4b8f8ac700 1 -- 172.21.15.88:0/3326128104 shutdown_connections 2023-07-14T17:43:05.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.085+0000 7f4b8f8ac700 1 -- 172.21.15.88:0/3326128104 wait complete. 2023-07-14T17:43:05.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8f8ac700 1 Processor -- start 2023-07-14T17:43:05.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8f8ac700 1 -- start start 2023-07-14T17:43:05.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8f8ac700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b880feb50 0x7f4b881946f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:05.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8f8ac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b880ffd40 0x7f4b88194c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:05.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8f8ac700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b881006d0 0x7f4b8819bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:05.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8f8ac700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8819c270 con 0x7f4b880ffd40 2023-07-14T17:43:05.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8f8ac700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8819c3b0 con 0x7f4b881006d0 2023-07-14T17:43:05.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8f8ac700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8819c540 con 0x7f4b880feb50 2023-07-14T17:43:05.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8ce47700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b880ffd40 0x7f4b88194c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:05.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8ce47700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b880ffd40 0x7f4b88194c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:41996/0 (socket says 172.21.15.88:41996) 2023-07-14T17:43:05.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8ce47700 1 -- 172.21.15.88:0/3425644091 learned_addr learned my addr 172.21.15.88:0/3425644091 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:05.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8d648700 1 --2- 172.21.15.88:0/3425644091 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b880feb50 0x7f4b881946f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:05.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8ce47700 1 -- 172.21.15.88:0/3425644091 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b880feb50 msgr2=0x7f4b881946f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:05.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8ce47700 1 --2- 172.21.15.88:0/3425644091 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b880feb50 0x7f4b881946f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8de49700 1 --2- 172.21.15.88:0/3425644091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b881006d0 0x7f4b8819bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:05.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8ce47700 1 -- 172.21.15.88:0/3425644091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b881006d0 msgr2=0x7f4b8819bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:05.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8ce47700 1 --2- 172.21.15.88:0/3425644091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b881006d0 0x7f4b8819bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8ce47700 1 -- 172.21.15.88:0/3425644091 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b840095f0 con 0x7f4b880ffd40 2023-07-14T17:43:05.078 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.086+0000 7f4b8ce47700 1 --2- 172.21.15.88:0/3425644091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b880ffd40 0x7f4b88194c30 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4b84003a50 tx=0x7f4b84009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:05.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.087+0000 7f4b7e7fc700 1 -- 172.21.15.88:0/3425644091 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8401d070 con 0x7f4b880ffd40 2023-07-14T17:43:05.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.087+0000 7f4b8f8ac700 1 -- 172.21.15.88:0/3425644091 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8819c7c0 con 0x7f4b880ffd40 2023-07-14T17:43:05.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.087+0000 7f4b7e7fc700 1 -- 172.21.15.88:0/3425644091 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4b84005a00 con 0x7f4b880ffd40 2023-07-14T17:43:05.079 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.087+0000 7f4b7e7fc700 1 -- 172.21.15.88:0/3425644091 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b84005020 con 0x7f4b880ffd40 2023-07-14T17:43:05.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.087+0000 7f4b8f8ac700 1 -- 172.21.15.88:0/3425644091 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8819cc10 con 0x7f4b880ffd40 2023-07-14T17:43:05.080 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.087+0000 7f4b8f8ac700 1 -- 172.21.15.88:0/3425644091 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b881060c0 con 0x7f4b880ffd40 2023-07-14T17:43:05.081 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.089+0000 7f4b7e7fc700 1 -- 172.21.15.88:0/3425644091 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4b840051a0 con 0x7f4b880ffd40 2023-07-14T17:43:05.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.089+0000 7f4b7e7fc700 1 --2- 172.21.15.88:0/3425644091 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4b740a1220 0x7f4b740a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:05.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.089+0000 7f4b7e7fc700 1 -- 172.21.15.88:0/3425644091 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4b840ce1c0 con 0x7f4b880ffd40 2023-07-14T17:43:05.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.089+0000 7f4b8d648700 1 --2- 172.21.15.88:0/3425644091 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4b740a1220 0x7f4b740a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:05.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.090+0000 7f4b8d648700 1 --2- 172.21.15.88:0/3425644091 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4b740a1220 0x7f4b740a36d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4b880ffba0 tx=0x7f4b7800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:05.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.093+0000 7f4b7e7fc700 1 -- 172.21.15.88:0/3425644091 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4b84097650 con 0x7f4b880ffd40 2023-07-14T17:43:05.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:04 smithi088 ceph-mon[127465]: pgmap v266: 129 pgs: 129 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 5 op/s 2023-07-14T17:43:05.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:04 smithi088 ceph-mon[127465]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:43:05.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:04 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] up:active 2023-07-14T17:43:05.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:04 smithi088 ceph-mon[127465]: fsmap cephfs:5 7 up:standby 5 up:active 2023-07-14T17:43:05.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:04 smithi088 ceph-mon[127465]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T17:43:05.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:04 smithi088 ceph-mon[127465]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-14T17:43:05.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:04 smithi107 ceph-mon[128999]: pgmap v266: 129 pgs: 129 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 5 op/s 2023-07-14T17:43:05.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:04 smithi107 ceph-mon[128999]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:43:05.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:04 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] up:active 2023-07-14T17:43:05.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:04 smithi107 ceph-mon[128999]: fsmap cephfs:5 7 up:standby 5 up:active 2023-07-14T17:43:05.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:04 smithi107 ceph-mon[128999]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T17:43:05.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:04 smithi107 ceph-mon[128999]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-14T17:43:05.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.329+0000 7f4b8f8ac700 1 -- 172.21.15.88:0/3425644091 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b8802c8e0 con 0x7f4b880ffd40 2023-07-14T17:43:05.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.330+0000 7f4b7e7fc700 1 -- 172.21.15.88:0/3425644091 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10885 (secure 0 0 0) 0x7f4b8409ac70 con 0x7f4b880ffd40 2023-07-14T17:43:05.312 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:43:05.312 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T17:42:42.314442+0000","modified":"2023-07-14T17:43:04.774381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24473,"mds_2":24469,"mds_3":24454,"mds_4":24449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6839/3046592155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3046592155},{"type":"v1","addr":"172.21.15.88:6839","nonce":3046592155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6841/2509502918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6840","nonce":2509502918},{"type":"v1","addr":"172.21.15.88:6841","nonce":2509502918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6835/2332025068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":2332025068},{"type":"v1","addr":"172.21.15.178:6835","nonce":2332025068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24449":{"gid":24449,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.107:6835/3062637672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3062637672},{"type":"v1","addr":"172.21.15.107:6835","nonce":3062637672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.178:6837/3950404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3950404084},{"type":"v1","addr":"172.21.15.178:6837","nonce":3950404084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.178:6839/2953415077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2953415077},{"type":"v1","addr":"172.21.15.178:6839","nonce":2953415077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/1152608615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1152608615},{"type":"v1","addr":"172.21.15.107:6837","nonce":1152608615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6839/31646548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":31646548},{"type":"v1","addr":"172.21.15.107:6839","nonce":31646548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T17:43:05.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.332+0000 7f4b8f8ac700 1 -- 172.21.15.88:0/3425644091 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4b740a1220 msgr2=0x7f4b740a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:05.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.332+0000 7f4b8f8ac700 1 --2- 172.21.15.88:0/3425644091 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4b740a1220 0x7f4b740a36d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4b880ffba0 tx=0x7f4b7800b410 comp rx=0 tx=0).stop 2023-07-14T17:43:05.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.332+0000 7f4b8f8ac700 1 -- 172.21.15.88:0/3425644091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b880ffd40 msgr2=0x7f4b88194c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:05.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.332+0000 7f4b8f8ac700 1 --2- 172.21.15.88:0/3425644091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b880ffd40 0x7f4b88194c30 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4b84003a50 tx=0x7f4b84009580 comp rx=0 tx=0).stop 2023-07-14T17:43:05.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.332+0000 7f4b8f8ac700 1 -- 172.21.15.88:0/3425644091 shutdown_connections 2023-07-14T17:43:05.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.332+0000 7f4b8f8ac700 1 --2- 172.21.15.88:0/3425644091 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4b881006d0 0x7f4b8819bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.332+0000 7f4b8f8ac700 1 --2- 172.21.15.88:0/3425644091 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4b880feb50 0x7f4b881946f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.332+0000 7f4b8f8ac700 1 --2- 172.21.15.88:0/3425644091 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4b740a1220 0x7f4b740a36d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.332+0000 7f4b8f8ac700 1 --2- 172.21.15.88:0/3425644091 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4b880ffd40 0x7f4b88194c30 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.332+0000 7f4b8f8ac700 1 -- 172.21.15.88:0/3425644091 >> 172.21.15.88:0/3425644091 conn(0x7f4b880fa150 msgr2=0x7f4b88103200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:05.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.332+0000 7f4b8f8ac700 1 -- 172.21.15.88:0/3425644091 shutdown_connections 2023-07-14T17:43:05.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.332+0000 7f4b8f8ac700 1 -- 172.21.15.88:0/3425644091 wait complete. 2023-07-14T17:43:05.316 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 27 2023-07-14T17:43:05.326 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:43:05.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:04 smithi178 ceph-mon[129368]: pgmap v266: 129 pgs: 129 active+clean; 582 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 5 op/s 2023-07-14T17:43:05.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:04 smithi178 ceph-mon[129368]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:43:05.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:04 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] up:active 2023-07-14T17:43:05.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:04 smithi178 ceph-mon[129368]: fsmap cephfs:5 7 up:standby 5 up:active 2023-07-14T17:43:05.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:04 smithi178 ceph-mon[129368]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T17:43:05.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:04 smithi178 ceph-mon[129368]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-14T17:43:05.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.464+0000 7f0b41a68700 1 -- 172.21.15.88:0/3908252634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b3c100740 msgr2=0x7f0b3c102b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:05.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.464+0000 7f0b41a68700 1 --2- 172.21.15.88:0/3908252634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b3c100740 0x7f0b3c102b20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c009990 tx=0x7f0b2c009ca0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.464+0000 7f0b41a68700 1 -- 172.21.15.88:0/3908252634 shutdown_connections 2023-07-14T17:43:05.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.464+0000 7f0b41a68700 1 --2- 172.21.15.88:0/3908252634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b3c105980 0x7f0b3c107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.464+0000 7f0b41a68700 1 --2- 172.21.15.88:0/3908252634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b3c100740 0x7f0b3c102b20 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.464+0000 7f0b41a68700 1 --2- 172.21.15.88:0/3908252634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b3c103060 0x7f0b3c105440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.464+0000 7f0b41a68700 1 -- 172.21.15.88:0/3908252634 >> 172.21.15.88:0/3908252634 conn(0x7f0b3c0fa100 msgr2=0x7f0b3c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.465+0000 7f0b41a68700 1 -- 172.21.15.88:0/3908252634 shutdown_connections 2023-07-14T17:43:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.465+0000 7f0b41a68700 1 -- 172.21.15.88:0/3908252634 wait complete. 2023-07-14T17:43:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.465+0000 7f0b41a68700 1 Processor -- start 2023-07-14T17:43:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.465+0000 7f0b41a68700 1 -- start start 2023-07-14T17:43:05.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.465+0000 7f0b41a68700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b3c100740 0x7f0b3c1958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:05.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.465+0000 7f0b41a68700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b3c103060 0x7f0b3c195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:05.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.465+0000 7f0b41a68700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b3c105980 0x7f0b3c19ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:05.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.465+0000 7f0b41a68700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c19d420 con 0x7f0b3c105980 2023-07-14T17:43:05.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.465+0000 7f0b41a68700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c19d560 con 0x7f0b3c103060 2023-07-14T17:43:05.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.465+0000 7f0b41a68700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c19d6f0 con 0x7f0b3c100740 2023-07-14T17:43:05.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.466+0000 7f0b3bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b3c105980 0x7f0b3c19ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:05.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.466+0000 7f0b3bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b3c105980 0x7f0b3c19ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42012/0 (socket says 172.21.15.88:42012) 2023-07-14T17:43:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.466+0000 7f0b3b7fe700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b3c100740 0x7f0b3c1958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.466+0000 7f0b3bfff700 1 -- 172.21.15.88:0/3004495923 learned_addr learned my addr 172.21.15.88:0/3004495923 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.466+0000 7f0b3affd700 1 --2- 172.21.15.88:0/3004495923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b3c103060 0x7f0b3c195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.466+0000 7f0b3bfff700 1 -- 172.21.15.88:0/3004495923 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b3c100740 msgr2=0x7f0b3c1958a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.466+0000 7f0b3bfff700 1 --2- 172.21.15.88:0/3004495923 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b3c100740 0x7f0b3c1958a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.452 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.466+0000 7f0b3bfff700 1 -- 172.21.15.88:0/3004495923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b3c103060 msgr2=0x7f0b3c195de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:05.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.466+0000 7f0b3bfff700 1 --2- 172.21.15.88:0/3004495923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b3c103060 0x7f0b3c195de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.466+0000 7f0b3bfff700 1 -- 172.21.15.88:0/3004495923 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2c0095f0 con 0x7f0b3c105980 2023-07-14T17:43:05.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.466+0000 7f0b3bfff700 1 --2- 172.21.15.88:0/3004495923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b3c105980 0x7f0b3c19ce50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0b3400ada0 tx=0x7f0b3400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:05.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.466+0000 7f0b38ff9700 1 -- 172.21.15.88:0/3004495923 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3400f070 con 0x7f0b3c105980 2023-07-14T17:43:05.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.466+0000 7f0b38ff9700 1 -- 172.21.15.88:0/3004495923 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0b34007b30 con 0x7f0b3c105980 2023-07-14T17:43:05.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.466+0000 7f0b38ff9700 1 -- 172.21.15.88:0/3004495923 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b34005280 con 0x7f0b3c105980 2023-07-14T17:43:05.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.466+0000 7f0b41a68700 1 -- 172.21.15.88:0/3004495923 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3c19d9d0 con 0x7f0b3c105980 2023-07-14T17:43:05.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.466+0000 7f0b41a68700 1 -- 172.21.15.88:0/3004495923 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3c19de90 con 0x7f0b3c105980 2023-07-14T17:43:05.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.467+0000 7f0b41a68700 1 -- 172.21.15.88:0/3004495923 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3c0fbc70 con 0x7f0b3c105980 2023-07-14T17:43:05.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.468+0000 7f0b38ff9700 1 -- 172.21.15.88:0/3004495923 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f0b3400a090 con 0x7f0b3c105980 2023-07-14T17:43:05.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.469+0000 7f0b38ff9700 1 --2- 172.21.15.88:0/3004495923 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0b240a12a0 0x7f0b240a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:05.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.469+0000 7f0b3b7fe700 1 --2- 172.21.15.88:0/3004495923 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0b240a12a0 0x7f0b240a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:05.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.469+0000 7f0b38ff9700 1 -- 172.21.15.88:0/3004495923 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0b340cc730 con 0x7f0b3c105980 2023-07-14T17:43:05.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.469+0000 7f0b3b7fe700 1 --2- 172.21.15.88:0/3004495923 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0b240a12a0 0x7f0b240a3750 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00b5c0 tx=0x7f0b2c009f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:05.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.472+0000 7f0b38ff9700 1 -- 172.21.15.88:0/3004495923 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0b34095bc0 con 0x7f0b3c105980 2023-07-14T17:43:05.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.661+0000 7f0b41a68700 1 -- 172.21.15.88:0/3004495923 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b3c02cb30 con 0x7f0b3c105980 2023-07-14T17:43:05.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.662+0000 7f0b38ff9700 1 -- 172.21.15.88:0/3004495923 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23985 (secure 0 0 0) 0x7f0b340991e0 con 0x7f0b3c105980 2023-07-14T17:43:05.645 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:43:05.645 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":86,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:42:46.934364+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:43:05.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.664+0000 7f0b41a68700 1 -- 172.21.15.88:0/3004495923 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0b240a12a0 msgr2=0x7f0b240a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:05.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.664+0000 7f0b41a68700 1 --2- 172.21.15.88:0/3004495923 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0b240a12a0 0x7f0b240a3750 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00b5c0 tx=0x7f0b2c009f00 comp rx=0 tx=0).stop 2023-07-14T17:43:05.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.664+0000 7f0b41a68700 1 -- 172.21.15.88:0/3004495923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b3c105980 msgr2=0x7f0b3c19ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:05.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.664+0000 7f0b41a68700 1 --2- 172.21.15.88:0/3004495923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b3c105980 0x7f0b3c19ce50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0b3400ada0 tx=0x7f0b3400c330 comp rx=0 tx=0).stop 2023-07-14T17:43:05.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.664+0000 7f0b41a68700 1 -- 172.21.15.88:0/3004495923 shutdown_connections 2023-07-14T17:43:05.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.664+0000 7f0b41a68700 1 --2- 172.21.15.88:0/3004495923 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0b240a12a0 0x7f0b240a3750 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.664+0000 7f0b41a68700 1 --2- 172.21.15.88:0/3004495923 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0b3c105980 0x7f0b3c19ce50 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.664+0000 7f0b41a68700 1 --2- 172.21.15.88:0/3004495923 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0b3c100740 0x7f0b3c1958a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.664+0000 7f0b41a68700 1 --2- 172.21.15.88:0/3004495923 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0b3c103060 0x7f0b3c195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.664+0000 7f0b41a68700 1 -- 172.21.15.88:0/3004495923 >> 172.21.15.88:0/3004495923 conn(0x7f0b3c0fa100 msgr2=0x7f0b3c1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:05.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.665+0000 7f0b41a68700 1 -- 172.21.15.88:0/3004495923 shutdown_connections 2023-07-14T17:43:05.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.665+0000 7f0b41a68700 1 -- 172.21.15.88:0/3004495923 wait complete. 2023-07-14T17:43:05.660 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-07-14T17:43:05.660 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-14T17:43:05.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.800+0000 7f3ff153b700 1 -- 172.21.15.88:0/229746409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fec0feb00 msgr2=0x7f3fec0fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:05.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.800+0000 7f3ff153b700 1 --2- 172.21.15.88:0/229746409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fec0feb00 0x7f3fec0fef10 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc009990 tx=0x7f3fdc009ca0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.800+0000 7f3ff153b700 1 -- 172.21.15.88:0/229746409 shutdown_connections 2023-07-14T17:43:05.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.800+0000 7f3ff153b700 1 --2- 172.21.15.88:0/229746409 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fec100680 0x7f3fec106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.800+0000 7f3ff153b700 1 --2- 172.21.15.88:0/229746409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fec0feb00 0x7f3fec0fef10 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.800+0000 7f3ff153b700 1 --2- 172.21.15.88:0/229746409 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fec0ffcf0 0x7f3fec100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.800+0000 7f3ff153b700 1 -- 172.21.15.88:0/229746409 >> 172.21.15.88:0/229746409 conn(0x7f3fec0fa100 msgr2=0x7f3fec0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:05.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.800+0000 7f3ff153b700 1 -- 172.21.15.88:0/229746409 shutdown_connections 2023-07-14T17:43:05.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.800+0000 7f3ff153b700 1 -- 172.21.15.88:0/229746409 wait complete. 2023-07-14T17:43:05.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.801+0000 7f3ff153b700 1 Processor -- start 2023-07-14T17:43:05.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.801+0000 7f3ff153b700 1 -- start start 2023-07-14T17:43:05.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.801+0000 7f3ff153b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fec0feb00 0x7f3fec194760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:05.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.801+0000 7f3ff153b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fec0ffcf0 0x7f3fec194ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:05.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.801+0000 7f3ff153b700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fec100680 0x7f3fec19bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:05.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.801+0000 7f3ff153b700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec19c2e0 con 0x7f3fec0feb00 2023-07-14T17:43:05.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3ff153b700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec19c420 con 0x7f3fec100680 2023-07-14T17:43:05.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3ff153b700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fec19c5b0 con 0x7f3fec0ffcf0 2023-07-14T17:43:05.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3feaffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fec0feb00 0x7f3fec194760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:05.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3feaffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fec0feb00 0x7f3fec194760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42022/0 (socket says 172.21.15.88:42022) 2023-07-14T17:43:05.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3feaffd700 1 -- 172.21.15.88:0/3377407009 learned_addr learned my addr 172.21.15.88:0/3377407009 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:05.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3fea7fc700 1 --2- 172.21.15.88:0/3377407009 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fec0ffcf0 0x7f3fec194ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:05.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3feb7fe700 1 --2- 172.21.15.88:0/3377407009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fec100680 0x7f3fec19bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:05.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3feaffd700 1 -- 172.21.15.88:0/3377407009 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fec0ffcf0 msgr2=0x7f3fec194ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:05.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3feaffd700 1 --2- 172.21.15.88:0/3377407009 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fec0ffcf0 0x7f3fec194ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3feaffd700 1 -- 172.21.15.88:0/3377407009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fec100680 msgr2=0x7f3fec19bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:05.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3feaffd700 1 --2- 172.21.15.88:0/3377407009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fec100680 0x7f3fec19bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3feaffd700 1 -- 172.21.15.88:0/3377407009 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fdc0095f0 con 0x7f3fec0feb00 2023-07-14T17:43:05.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3feb7fe700 1 --2- 172.21.15.88:0/3377407009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fec100680 0x7f3fec19bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:43:05.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3feaffd700 1 --2- 172.21.15.88:0/3377407009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fec0feb00 0x7f3fec194760 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc00b5c0 tx=0x7f3fdc005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:05.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3fd3fff700 1 -- 172.21.15.88:0/3377407009 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc01d070 con 0x7f3fec0feb00 2023-07-14T17:43:05.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3fd3fff700 1 -- 172.21.15.88:0/3377407009 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3fdc004080 con 0x7f3fec0feb00 2023-07-14T17:43:05.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3ff153b700 1 -- 172.21.15.88:0/3377407009 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fec19c830 con 0x7f3fec0feb00 2023-07-14T17:43:05.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3fd3fff700 1 -- 172.21.15.88:0/3377407009 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc022cd0 con 0x7f3fec0feb00 2023-07-14T17:43:05.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.802+0000 7f3ff153b700 1 -- 172.21.15.88:0/3377407009 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fec19ccb0 con 0x7f3fec0feb00 2023-07-14T17:43:05.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.803+0000 7f3ff153b700 1 -- 172.21.15.88:0/3377407009 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fec10a180 con 0x7f3fec0feb00 2023-07-14T17:43:05.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.804+0000 7f3fd3fff700 1 -- 172.21.15.88:0/3377407009 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3fdc022470 con 0x7f3fec0feb00 2023-07-14T17:43:05.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.805+0000 7f3fd3fff700 1 --2- 172.21.15.88:0/3377407009 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3fd40a12a0 0x7f3fd40a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:05.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.805+0000 7f3fea7fc700 1 --2- 172.21.15.88:0/3377407009 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3fd40a12a0 0x7f3fd40a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:05.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.805+0000 7f3fd3fff700 1 -- 172.21.15.88:0/3377407009 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3fdc0ce8b0 con 0x7f3fec0feb00 2023-07-14T17:43:05.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.805+0000 7f3fea7fc700 1 --2- 172.21.15.88:0/3377407009 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3fd40a12a0 0x7f3fd40a3750 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0005950 tx=0x7f3fe00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:05.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.808+0000 7f3fd3fff700 1 -- 172.21.15.88:0/3377407009 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3fdc097d40 con 0x7f3fec0feb00 2023-07-14T17:43:05.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:05.996+0000 7f3ff153b700 1 -- 172.21.15.88:0/3377407009 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3fec060a80 con 0x7f3fd40a12a0 2023-07-14T17:43:05.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.009+0000 7f3fd3fff700 1 -- 172.21.15.88:0/3377407009 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+418193 (secure 0 0 0) 0x7f3fec060a80 con 0x7f3fd40a12a0 2023-07-14T17:43:05.996 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:43:05.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.012+0000 7f3ff153b700 1 -- 172.21.15.88:0/3377407009 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3fd40a12a0 msgr2=0x7f3fd40a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:05.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.012+0000 7f3ff153b700 1 --2- 172.21.15.88:0/3377407009 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3fd40a12a0 0x7f3fd40a3750 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0005950 tx=0x7f3fe00058e0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.012+0000 7f3ff153b700 1 -- 172.21.15.88:0/3377407009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fec0feb00 msgr2=0x7f3fec194760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:05.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.012+0000 7f3ff153b700 1 --2- 172.21.15.88:0/3377407009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fec0feb00 0x7f3fec194760 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc00b5c0 tx=0x7f3fdc005ef0 comp rx=0 tx=0).stop 2023-07-14T17:43:05.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.013+0000 7f3ff153b700 1 -- 172.21.15.88:0/3377407009 shutdown_connections 2023-07-14T17:43:05.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.013+0000 7f3ff153b700 1 --2- 172.21.15.88:0/3377407009 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3fec100680 0x7f3fec19bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:06.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.013+0000 7f3ff153b700 1 --2- 172.21.15.88:0/3377407009 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3fd40a12a0 0x7f3fd40a3750 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:06.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.013+0000 7f3ff153b700 1 --2- 172.21.15.88:0/3377407009 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3fec0feb00 0x7f3fec194760 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:06.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.013+0000 7f3ff153b700 1 --2- 172.21.15.88:0/3377407009 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3fec0ffcf0 0x7f3fec194ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:06.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.013+0000 7f3ff153b700 1 -- 172.21.15.88:0/3377407009 >> 172.21.15.88:0/3377407009 conn(0x7f3fec0fa100 msgr2=0x7f3fec0fc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:06.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.013+0000 7f3ff153b700 1 -- 172.21.15.88:0/3377407009 shutdown_connections 2023-07-14T17:43:06.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.013+0000 7f3ff153b700 1 -- 172.21.15.88:0/3377407009 wait complete. 2023-07-14T17:43:06.000 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-07-14T17:43:06.007 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2023-07-14T17:43:05.875883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":634167,"num_objects":84,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":147,"num_read_kb":163,"num_write":369,"num_write_kb":3038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":302,"ondisk_log_size":302,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337424,"kb_used_data":13224,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604400,"statfs":{"total":1151940427776,"available":1151594905600,"internally_reserved":0,"allocated":13541376,"data_stored":7432032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41437,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":21,"num_read_kb":54,"num_write":143,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003020"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.060009+0000","last_change":"2023-07-14T17:42:42.540396+0000","last_active":"2023-07-14T17:42:48.060009+0000","last_peered":"2023-07-14T17:42:48.060009+0000","last_clean":"2023-07-14T17:42:48.060009+0000","last_became_active":"2023-07-14T17:42:42.539732+0000","last_became_peered":"2023-07-14T17:42:42.539732+0000","last_unstale":"2023-07-14T17:42:48.060009+0000","last_undegraded":"2023-07-14T17:42:48.060009+0000","last_fullsized":"2023-07-14T17:42:48.060009+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:15:44.925121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313254+0000","last_change":"2023-07-14T17:42:42.532153+0000","last_active":"2023-07-14T17:42:48.313254+0000","last_peered":"2023-07-14T17:42:48.313254+0000","last_clean":"2023-07-14T17:42:48.313254+0000","last_became_active":"2023-07-14T17:42:42.531745+0000","last_became_peered":"2023-07-14T17:42:42.531745+0000","last_unstale":"2023-07-14T17:42:48.313254+0000","last_undegraded":"2023-07-14T17:42:48.313254+0000","last_fullsized":"2023-07-14T17:42:48.313254+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:55.889251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313184+0000","last_change":"2023-07-14T17:42:42.549497+0000","last_active":"2023-07-14T17:42:48.313184+0000","last_peered":"2023-07-14T17:42:48.313184+0000","last_clean":"2023-07-14T17:42:48.313184+0000","last_became_active":"2023-07-14T17:42:42.549201+0000","last_became_peered":"2023-07-14T17:42:42.549201+0000","last_unstale":"2023-07-14T17:42:48.313184+0000","last_undegraded":"2023-07-14T17:42:48.313184+0000","last_fullsized":"2023-07-14T17:42:48.313184+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:20:16.806348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542180+0000","last_change":"2023-07-14T17:42:42.543460+0000","last_active":"2023-07-14T17:42:47.542180+0000","last_peered":"2023-07-14T17:42:47.542180+0000","last_clean":"2023-07-14T17:42:47.542180+0000","last_became_active":"2023-07-14T17:42:42.543127+0000","last_became_peered":"2023-07-14T17:42:42.543127+0000","last_unstale":"2023-07-14T17:42:47.542180+0000","last_undegraded":"2023-07-14T17:42:47.542180+0000","last_fullsized":"2023-07-14T17:42:47.542180+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:40:04.077779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835392+0000","last_change":"2023-07-14T17:42:42.534242+0000","last_active":"2023-07-14T17:42:47.835392+0000","last_peered":"2023-07-14T17:42:47.835392+0000","last_clean":"2023-07-14T17:42:47.835392+0000","last_became_active":"2023-07-14T17:42:42.533536+0000","last_became_peered":"2023-07-14T17:42:42.533536+0000","last_unstale":"2023-07-14T17:42:47.835392+0000","last_undegraded":"2023-07-14T17:42:47.835392+0000","last_fullsized":"2023-07-14T17:42:47.835392+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:57:39.927450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313767+0000","last_change":"2023-07-14T17:42:42.538042+0000","last_active":"2023-07-14T17:42:48.313767+0000","last_peered":"2023-07-14T17:42:48.313767+0000","last_clean":"2023-07-14T17:42:48.313767+0000","last_became_active":"2023-07-14T17:42:42.537612+0000","last_became_peered":"2023-07-14T17:42:42.537612+0000","last_unstale":"2023-07-14T17:42:48.313767+0000","last_undegraded":"2023-07-14T17:42:48.313767+0000","last_fullsized":"2023-07-14T17:42:48.313767+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:18:57.774815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313672+0000","last_change":"2023-07-14T17:42:42.545318+0000","last_active":"2023-07-14T17:42:48.313672+0000","last_peered":"2023-07-14T17:42:48.313672+0000","last_clean":"2023-07-14T17:42:48.313672+0000","last_became_active":"2023-07-14T17:42:42.544984+0000","last_became_peered":"2023-07-14T17:42:42.544984+0000","last_unstale":"2023-07-14T17:42:48.313672+0000","last_undegraded":"2023-07-14T17:42:48.313672+0000","last_fullsized":"2023-07-14T17:42:48.313672+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:21:01.756500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539255+0000","last_change":"2023-07-14T17:42:42.532609+0000","last_active":"2023-07-14T17:42:47.539255+0000","last_peered":"2023-07-14T17:42:47.539255+0000","last_clean":"2023-07-14T17:42:47.539255+0000","last_became_active":"2023-07-14T17:42:42.532270+0000","last_became_peered":"2023-07-14T17:42:42.532270+0000","last_unstale":"2023-07-14T17:42:47.539255+0000","last_undegraded":"2023-07-14T17:42:47.539255+0000","last_fullsized":"2023-07-14T17:42:47.539255+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:20:00.572052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.315239+0000","last_change":"2023-07-14T17:42:42.548973+0000","last_active":"2023-07-14T17:42:48.315239+0000","last_peered":"2023-07-14T17:42:48.315239+0000","last_clean":"2023-07-14T17:42:48.315239+0000","last_became_active":"2023-07-14T17:42:42.548672+0000","last_became_peered":"2023-07-14T17:42:42.548672+0000","last_unstale":"2023-07-14T17:42:48.315239+0000","last_undegraded":"2023-07-14T17:42:48.315239+0000","last_fullsized":"2023-07-14T17:42:48.315239+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:52:48.887371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.315082+0000","last_change":"2023-07-14T17:42:42.540969+0000","last_active":"2023-07-14T17:42:48.315082+0000","last_peered":"2023-07-14T17:42:48.315082+0000","last_clean":"2023-07-14T17:42:48.315082+0000","last_became_active":"2023-07-14T17:42:42.540653+0000","last_became_peered":"2023-07-14T17:42:42.540653+0000","last_unstale":"2023-07-14T17:42:48.315082+0000","last_undegraded":"2023-07-14T17:42:48.315082+0000","last_fullsized":"2023-07-14T17:42:48.315082+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:18:49.023990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314444+0000","last_change":"2023-07-14T17:42:42.550016+0000","last_active":"2023-07-14T17:42:48.314444+0000","last_peered":"2023-07-14T17:42:48.314444+0000","last_clean":"2023-07-14T17:42:48.314444+0000","last_became_active":"2023-07-14T17:42:42.549653+0000","last_became_peered":"2023-07-14T17:42:42.549653+0000","last_unstale":"2023-07-14T17:42:48.314444+0000","last_undegraded":"2023-07-14T17:42:48.314444+0000","last_fullsized":"2023-07-14T17:42:48.314444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:31:15.876030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542109+0000","last_change":"2023-07-14T17:42:42.552105+0000","last_active":"2023-07-14T17:42:47.542109+0000","last_peered":"2023-07-14T17:42:47.542109+0000","last_clean":"2023-07-14T17:42:47.542109+0000","last_became_active":"2023-07-14T17:42:42.550830+0000","last_became_peered":"2023-07-14T17:42:42.550830+0000","last_unstale":"2023-07-14T17:42:47.542109+0000","last_undegraded":"2023-07-14T17:42:47.542109+0000","last_fullsized":"2023-07-14T17:42:47.542109+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:47:37.896520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313576+0000","last_change":"2023-07-14T17:42:42.534963+0000","last_active":"2023-07-14T17:42:48.313576+0000","last_peered":"2023-07-14T17:42:48.313576+0000","last_clean":"2023-07-14T17:42:48.313576+0000","last_became_active":"2023-07-14T17:42:42.534546+0000","last_became_peered":"2023-07-14T17:42:42.534546+0000","last_unstale":"2023-07-14T17:42:48.313576+0000","last_undegraded":"2023-07-14T17:42:48.313576+0000","last_fullsized":"2023-07-14T17:42:48.313576+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:01:37.492549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.312996+0000","last_change":"2023-07-14T17:42:42.530073+0000","last_active":"2023-07-14T17:42:48.312996+0000","last_peered":"2023-07-14T17:42:48.312996+0000","last_clean":"2023-07-14T17:42:48.312996+0000","last_became_active":"2023-07-14T17:42:42.529767+0000","last_became_peered":"2023-07-14T17:42:42.529767+0000","last_unstale":"2023-07-14T17:42:48.312996+0000","last_undegraded":"2023-07-14T17:42:48.312996+0000","last_fullsized":"2023-07-14T17:42:48.312996+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:13:05.025890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.312868+0000","last_change":"2023-07-14T17:42:42.548168+0000","last_active":"2023-07-14T17:42:48.312868+0000","last_peered":"2023-07-14T17:42:48.312868+0000","last_clean":"2023-07-14T17:42:48.312868+0000","last_became_active":"2023-07-14T17:42:42.547363+0000","last_became_peered":"2023-07-14T17:42:42.547363+0000","last_unstale":"2023-07-14T17:42:48.312868+0000","last_undegraded":"2023-07-14T17:42:48.312868+0000","last_fullsized":"2023-07-14T17:42:48.312868+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:25:15.031613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313218+0000","last_change":"2023-07-14T17:42:42.551042+0000","last_active":"2023-07-14T17:42:48.313218+0000","last_peered":"2023-07-14T17:42:48.313218+0000","last_clean":"2023-07-14T17:42:48.313218+0000","last_became_active":"2023-07-14T17:42:42.550730+0000","last_became_peered":"2023-07-14T17:42:42.550730+0000","last_unstale":"2023-07-14T17:42:48.313218+0000","last_undegraded":"2023-07-14T17:42:48.313218+0000","last_fullsized":"2023-07-14T17:42:48.313218+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:34:57.949192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059890+0000","last_change":"2023-07-14T17:42:42.540528+0000","last_active":"2023-07-14T17:42:48.059890+0000","last_peered":"2023-07-14T17:42:48.059890+0000","last_clean":"2023-07-14T17:42:48.059890+0000","last_became_active":"2023-07-14T17:42:42.540093+0000","last_became_peered":"2023-07-14T17:42:42.540093+0000","last_unstale":"2023-07-14T17:42:48.059890+0000","last_undegraded":"2023-07-14T17:42:48.059890+0000","last_fullsized":"2023-07-14T17:42:48.059890+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:03:51.839812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833464+0000","last_change":"2023-07-14T17:42:42.539512+0000","last_active":"2023-07-14T17:42:47.833464+0000","last_peered":"2023-07-14T17:42:47.833464+0000","last_clean":"2023-07-14T17:42:47.833464+0000","last_became_active":"2023-07-14T17:42:42.539248+0000","last_became_peered":"2023-07-14T17:42:42.539248+0000","last_unstale":"2023-07-14T17:42:47.833464+0000","last_undegraded":"2023-07-14T17:42:47.833464+0000","last_fullsized":"2023-07-14T17:42:47.833464+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:02:47.593908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123669+0000","last_change":"2023-07-14T17:42:42.531630+0000","last_active":"2023-07-14T17:42:48.123669+0000","last_peered":"2023-07-14T17:42:48.123669+0000","last_clean":"2023-07-14T17:42:48.123669+0000","last_became_active":"2023-07-14T17:42:42.531333+0000","last_became_peered":"2023-07-14T17:42:42.531333+0000","last_unstale":"2023-07-14T17:42:48.123669+0000","last_undegraded":"2023-07-14T17:42:48.123669+0000","last_fullsized":"2023-07-14T17:42:48.123669+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:45:40.106913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123071+0000","last_change":"2023-07-14T17:42:46.543062+0000","last_active":"2023-07-14T17:42:48.123071+0000","last_peered":"2023-07-14T17:42:48.123071+0000","last_clean":"2023-07-14T17:42:48.123071+0000","last_became_active":"2023-07-14T17:42:46.542189+0000","last_became_peered":"2023-07-14T17:42:46.542189+0000","last_unstale":"2023-07-14T17:42:48.123071+0000","last_undegraded":"2023-07-14T17:42:48.123071+0000","last_fullsized":"2023-07-14T17:42:48.123071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:42:27.430887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'11","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.761444+0000","last_change":"2023-07-14T17:42:41.914589+0000","last_active":"2023-07-14T17:43:02.761444+0000","last_peered":"2023-07-14T17:43:02.761444+0000","last_clean":"2023-07-14T17:43:02.761444+0000","last_became_active":"2023-07-14T17:42:41.914275+0000","last_became_peered":"2023-07-14T17:42:41.914275+0000","last_unstale":"2023-07-14T17:43:02.761444+0000","last_undegraded":"2023-07-14T17:43:02.761444+0000","last_fullsized":"2023-07-14T17:43:02.761444+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:09:10.487178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540020+0000","last_change":"2023-07-14T17:42:42.537207+0000","last_active":"2023-07-14T17:42:47.540020+0000","last_peered":"2023-07-14T17:42:47.540020+0000","last_clean":"2023-07-14T17:42:47.540020+0000","last_became_active":"2023-07-14T17:42:42.536879+0000","last_became_peered":"2023-07-14T17:42:42.536879+0000","last_unstale":"2023-07-14T17:42:47.540020+0000","last_undegraded":"2023-07-14T17:42:47.540020+0000","last_fullsized":"2023-07-14T17:42:47.540020+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:09:29.936504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.541631+0000","last_change":"2023-07-14T17:42:46.542500+0000","last_active":"2023-07-14T17:42:47.541631+0000","last_peered":"2023-07-14T17:42:47.541631+0000","last_clean":"2023-07-14T17:42:47.541631+0000","last_became_active":"2023-07-14T17:42:46.541932+0000","last_became_peered":"2023-07-14T17:42:46.541932+0000","last_unstale":"2023-07-14T17:42:47.541631+0000","last_undegraded":"2023-07-14T17:42:47.541631+0000","last_fullsized":"2023-07-14T17:42:47.541631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:07:53.233373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540068+0000","last_change":"2023-07-14T17:42:41.918169+0000","last_active":"2023-07-14T17:42:47.540068+0000","last_peered":"2023-07-14T17:42:47.540068+0000","last_clean":"2023-07-14T17:42:47.540068+0000","last_became_active":"2023-07-14T17:42:41.917124+0000","last_became_peered":"2023-07-14T17:42:41.917124+0000","last_unstale":"2023-07-14T17:42:47.540068+0000","last_undegraded":"2023-07-14T17:42:47.540068+0000","last_fullsized":"2023-07-14T17:42:47.540068+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:23:35.325653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313826+0000","last_change":"2023-07-14T17:42:42.539265+0000","last_active":"2023-07-14T17:42:48.313826+0000","last_peered":"2023-07-14T17:42:48.313826+0000","last_clean":"2023-07-14T17:42:48.313826+0000","last_became_active":"2023-07-14T17:42:42.538933+0000","last_became_peered":"2023-07-14T17:42:42.538933+0000","last_unstale":"2023-07-14T17:42:48.313826+0000","last_undegraded":"2023-07-14T17:42:48.313826+0000","last_fullsized":"2023-07-14T17:42:48.313826+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:08.277561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542003+0000","last_change":"2023-07-14T17:42:46.550194+0000","last_active":"2023-07-14T17:42:47.542003+0000","last_peered":"2023-07-14T17:42:47.542003+0000","last_clean":"2023-07-14T17:42:47.542003+0000","last_became_active":"2023-07-14T17:42:46.549693+0000","last_became_peered":"2023-07-14T17:42:46.549693+0000","last_unstale":"2023-07-14T17:42:47.542003+0000","last_undegraded":"2023-07-14T17:42:47.542003+0000","last_fullsized":"2023-07-14T17:42:47.542003+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:15.448341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'7","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.762657+0000","last_change":"2023-07-14T17:42:41.918769+0000","last_active":"2023-07-14T17:43:02.762657+0000","last_peered":"2023-07-14T17:43:02.762657+0000","last_clean":"2023-07-14T17:43:02.762657+0000","last_became_active":"2023-07-14T17:42:41.918573+0000","last_became_peered":"2023-07-14T17:42:41.918573+0000","last_unstale":"2023-07-14T17:43:02.762657+0000","last_undegraded":"2023-07-14T17:43:02.762657+0000","last_fullsized":"2023-07-14T17:43:02.762657+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:19:21.409733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.315010+0000","last_change":"2023-07-14T17:42:42.543846+0000","last_active":"2023-07-14T17:42:48.315010+0000","last_peered":"2023-07-14T17:42:48.315010+0000","last_clean":"2023-07-14T17:42:48.315010+0000","last_became_active":"2023-07-14T17:42:42.543528+0000","last_became_peered":"2023-07-14T17:42:42.543528+0000","last_unstale":"2023-07-14T17:42:48.315010+0000","last_undegraded":"2023-07-14T17:42:48.315010+0000","last_fullsized":"2023-07-14T17:42:48.315010+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:12.999288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.541773+0000","last_change":"2023-07-14T17:42:46.547799+0000","last_active":"2023-07-14T17:42:47.541773+0000","last_peered":"2023-07-14T17:42:47.541773+0000","last_clean":"2023-07-14T17:42:47.541773+0000","last_became_active":"2023-07-14T17:42:46.547381+0000","last_became_peered":"2023-07-14T17:42:46.547381+0000","last_unstale":"2023-07-14T17:42:47.541773+0000","last_undegraded":"2023-07-14T17:42:47.541773+0000","last_fullsized":"2023-07-14T17:42:47.541773+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:46:15.220406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835608+0000","last_change":"2023-07-14T17:42:41.914911+0000","last_active":"2023-07-14T17:42:47.835608+0000","last_peered":"2023-07-14T17:42:47.835608+0000","last_clean":"2023-07-14T17:42:47.835608+0000","last_became_active":"2023-07-14T17:42:41.914607+0000","last_became_peered":"2023-07-14T17:42:41.914607+0000","last_unstale":"2023-07-14T17:42:47.835608+0000","last_undegraded":"2023-07-14T17:42:47.835608+0000","last_fullsized":"2023-07-14T17:42:47.835608+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:45.771966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540287+0000","last_change":"2023-07-14T17:42:42.548866+0000","last_active":"2023-07-14T17:42:47.540287+0000","last_peered":"2023-07-14T17:42:47.540287+0000","last_clean":"2023-07-14T17:42:47.540287+0000","last_became_active":"2023-07-14T17:42:42.548562+0000","last_became_peered":"2023-07-14T17:42:42.548562+0000","last_unstale":"2023-07-14T17:42:47.540287+0000","last_undegraded":"2023-07-14T17:42:47.540287+0000","last_fullsized":"2023-07-14T17:42:47.540287+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:56:04.312858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123537+0000","last_change":"2023-07-14T17:42:46.538879+0000","last_active":"2023-07-14T17:42:48.123537+0000","last_peered":"2023-07-14T17:42:48.123537+0000","last_clean":"2023-07-14T17:42:48.123537+0000","last_became_active":"2023-07-14T17:42:46.538476+0000","last_became_peered":"2023-07-14T17:42:46.538476+0000","last_unstale":"2023-07-14T17:42:48.123537+0000","last_undegraded":"2023-07-14T17:42:48.123537+0000","last_fullsized":"2023-07-14T17:42:48.123537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:40:22.866252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.786373+0000","last_change":"2023-07-14T17:42:41.917215+0000","last_active":"2023-07-14T17:43:04.786373+0000","last_peered":"2023-07-14T17:43:04.786373+0000","last_clean":"2023-07-14T17:43:04.786373+0000","last_became_active":"2023-07-14T17:42:41.916640+0000","last_became_peered":"2023-07-14T17:42:41.916640+0000","last_unstale":"2023-07-14T17:43:04.786373+0000","last_undegraded":"2023-07-14T17:43:04.786373+0000","last_fullsized":"2023-07-14T17:43:04.786373+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:37:10.013820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835389+0000","last_change":"2023-07-14T17:42:42.534347+0000","last_active":"2023-07-14T17:42:47.835389+0000","last_peered":"2023-07-14T17:42:47.835389+0000","last_clean":"2023-07-14T17:42:47.835389+0000","last_became_active":"2023-07-14T17:42:42.533958+0000","last_became_peered":"2023-07-14T17:42:42.533958+0000","last_unstale":"2023-07-14T17:42:47.835389+0000","last_undegraded":"2023-07-14T17:42:47.835389+0000","last_fullsized":"2023-07-14T17:42:47.835389+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:52:39.908230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123386+0000","last_change":"2023-07-14T17:42:46.543144+0000","last_active":"2023-07-14T17:42:48.123386+0000","last_peered":"2023-07-14T17:42:48.123386+0000","last_clean":"2023-07-14T17:42:48.123386+0000","last_became_active":"2023-07-14T17:42:46.542311+0000","last_became_peered":"2023-07-14T17:42:46.542311+0000","last_unstale":"2023-07-14T17:42:48.123386+0000","last_undegraded":"2023-07-14T17:42:48.123386+0000","last_fullsized":"2023-07-14T17:42:48.123386+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:05:15.245426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.762300+0000","last_change":"2023-07-14T17:42:41.916174+0000","last_active":"2023-07-14T17:43:02.762300+0000","last_peered":"2023-07-14T17:43:02.762300+0000","last_clean":"2023-07-14T17:43:02.762300+0000","last_became_active":"2023-07-14T17:42:41.915867+0000","last_became_peered":"2023-07-14T17:42:41.915867+0000","last_unstale":"2023-07-14T17:43:02.762300+0000","last_undegraded":"2023-07-14T17:43:02.762300+0000","last_fullsized":"2023-07-14T17:43:02.762300+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:43.000811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540038+0000","last_change":"2023-07-14T17:42:42.547064+0000","last_active":"2023-07-14T17:42:47.540038+0000","last_peered":"2023-07-14T17:42:47.540038+0000","last_clean":"2023-07-14T17:42:47.540038+0000","last_became_active":"2023-07-14T17:42:42.546777+0000","last_became_peered":"2023-07-14T17:42:42.546777+0000","last_unstale":"2023-07-14T17:42:47.540038+0000","last_undegraded":"2023-07-14T17:42:47.540038+0000","last_fullsized":"2023-07-14T17:42:47.540038+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:39:05.720215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059852+0000","last_change":"2023-07-14T17:42:46.548647+0000","last_active":"2023-07-14T17:42:48.059852+0000","last_peered":"2023-07-14T17:42:48.059852+0000","last_clean":"2023-07-14T17:42:48.059852+0000","last_became_active":"2023-07-14T17:42:46.548142+0000","last_became_peered":"2023-07-14T17:42:46.548142+0000","last_unstale":"2023-07-14T17:42:48.059852+0000","last_undegraded":"2023-07-14T17:42:48.059852+0000","last_fullsized":"2023-07-14T17:42:48.059852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:52:44.965835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:01.756003+0000","last_change":"2023-07-14T17:42:41.911675+0000","last_active":"2023-07-14T17:43:01.756003+0000","last_peered":"2023-07-14T17:43:01.756003+0000","last_clean":"2023-07-14T17:43:01.756003+0000","last_became_active":"2023-07-14T17:42:41.911299+0000","last_became_peered":"2023-07-14T17:42:41.911299+0000","last_unstale":"2023-07-14T17:43:01.756003+0000","last_undegraded":"2023-07-14T17:43:01.756003+0000","last_fullsized":"2023-07-14T17:43:01.756003+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:09.880954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539414+0000","last_change":"2023-07-14T17:42:42.536211+0000","last_active":"2023-07-14T17:42:47.539414+0000","last_peered":"2023-07-14T17:42:47.539414+0000","last_clean":"2023-07-14T17:42:47.539414+0000","last_became_active":"2023-07-14T17:42:42.535763+0000","last_became_peered":"2023-07-14T17:42:42.535763+0000","last_unstale":"2023-07-14T17:42:47.539414+0000","last_undegraded":"2023-07-14T17:42:47.539414+0000","last_fullsized":"2023-07-14T17:42:47.539414+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:49:14.800525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313179+0000","last_change":"2023-07-14T17:42:46.540917+0000","last_active":"2023-07-14T17:42:48.313179+0000","last_peered":"2023-07-14T17:42:48.313179+0000","last_clean":"2023-07-14T17:42:48.313179+0000","last_became_active":"2023-07-14T17:42:46.540575+0000","last_became_peered":"2023-07-14T17:42:46.540575+0000","last_unstale":"2023-07-14T17:42:48.313179+0000","last_undegraded":"2023-07-14T17:42:48.313179+0000","last_fullsized":"2023-07-14T17:42:48.313179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:02:20.835521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.771356+0000","last_change":"2023-07-14T17:42:41.917646+0000","last_active":"2023-07-14T17:43:03.771356+0000","last_peered":"2023-07-14T17:43:03.771356+0000","last_clean":"2023-07-14T17:43:03.771356+0000","last_became_active":"2023-07-14T17:42:41.917331+0000","last_became_peered":"2023-07-14T17:42:41.917331+0000","last_unstale":"2023-07-14T17:43:03.771356+0000","last_undegraded":"2023-07-14T17:43:03.771356+0000","last_fullsized":"2023-07-14T17:43:03.771356+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:41:16.425142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833094+0000","last_change":"2023-07-14T17:42:42.526696+0000","last_active":"2023-07-14T17:42:47.833094+0000","last_peered":"2023-07-14T17:42:47.833094+0000","last_clean":"2023-07-14T17:42:47.833094+0000","last_became_active":"2023-07-14T17:42:42.526353+0000","last_became_peered":"2023-07-14T17:42:42.526353+0000","last_unstale":"2023-07-14T17:42:47.833094+0000","last_undegraded":"2023-07-14T17:42:47.833094+0000","last_fullsized":"2023-07-14T17:42:47.833094+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:01:54.943614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539428+0000","last_change":"2023-07-14T17:42:46.545460+0000","last_active":"2023-07-14T17:42:47.539428+0000","last_peered":"2023-07-14T17:42:47.539428+0000","last_clean":"2023-07-14T17:42:47.539428+0000","last_became_active":"2023-07-14T17:42:46.544897+0000","last_became_peered":"2023-07-14T17:42:46.544897+0000","last_unstale":"2023-07-14T17:42:47.539428+0000","last_undegraded":"2023-07-14T17:42:47.539428+0000","last_fullsized":"2023-07-14T17:42:47.539428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:18:28.628297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.782895+0000","last_change":"2023-07-14T17:42:41.912266+0000","last_active":"2023-07-14T17:43:04.782895+0000","last_peered":"2023-07-14T17:43:04.782895+0000","last_clean":"2023-07-14T17:43:04.782895+0000","last_became_active":"2023-07-14T17:42:41.911944+0000","last_became_peered":"2023-07-14T17:42:41.911944+0000","last_unstale":"2023-07-14T17:43:04.782895+0000","last_undegraded":"2023-07-14T17:43:04.782895+0000","last_fullsized":"2023-07-14T17:43:04.782895+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:10:15.152078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.315346+0000","last_change":"2023-07-14T17:42:42.548609+0000","last_active":"2023-07-14T17:42:48.315346+0000","last_peered":"2023-07-14T17:42:48.315346+0000","last_clean":"2023-07-14T17:42:48.315346+0000","last_became_active":"2023-07-14T17:42:42.548253+0000","last_became_peered":"2023-07-14T17:42:42.548253+0000","last_unstale":"2023-07-14T17:42:48.315346+0000","last_undegraded":"2023-07-14T17:42:48.315346+0000","last_fullsized":"2023-07-14T17:42:48.315346+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:02:49.871094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833114+0000","last_change":"2023-07-14T17:42:46.547324+0000","last_active":"2023-07-14T17:42:47.833114+0000","last_peered":"2023-07-14T17:42:47.833114+0000","last_clean":"2023-07-14T17:42:47.833114+0000","last_became_active":"2023-07-14T17:42:46.546876+0000","last_became_peered":"2023-07-14T17:42:46.546876+0000","last_unstale":"2023-07-14T17:42:47.833114+0000","last_undegraded":"2023-07-14T17:42:47.833114+0000","last_fullsized":"2023-07-14T17:42:47.833114+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:30:13.764106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:00.743584+0000","last_change":"2023-07-14T17:42:41.919148+0000","last_active":"2023-07-14T17:43:00.743584+0000","last_peered":"2023-07-14T17:43:00.743584+0000","last_clean":"2023-07-14T17:43:00.743584+0000","last_became_active":"2023-07-14T17:42:41.918842+0000","last_became_peered":"2023-07-14T17:42:41.918842+0000","last_unstale":"2023-07-14T17:43:00.743584+0000","last_undegraded":"2023-07-14T17:43:00.743584+0000","last_fullsized":"2023-07-14T17:43:00.743584+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:04:34.077766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313896+0000","last_change":"2023-07-14T17:42:42.548081+0000","last_active":"2023-07-14T17:42:48.313896+0000","last_peered":"2023-07-14T17:42:48.313896+0000","last_clean":"2023-07-14T17:42:48.313896+0000","last_became_active":"2023-07-14T17:42:42.547876+0000","last_became_peered":"2023-07-14T17:42:42.547876+0000","last_unstale":"2023-07-14T17:42:48.313896+0000","last_undegraded":"2023-07-14T17:42:48.313896+0000","last_fullsized":"2023-07-14T17:42:48.313896+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:53:08.262819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542183+0000","last_change":"2023-07-14T17:42:46.550516+0000","last_active":"2023-07-14T17:42:47.542183+0000","last_peered":"2023-07-14T17:42:47.542183+0000","last_clean":"2023-07-14T17:42:47.542183+0000","last_became_active":"2023-07-14T17:42:46.549588+0000","last_became_peered":"2023-07-14T17:42:46.549588+0000","last_unstale":"2023-07-14T17:42:47.542183+0000","last_undegraded":"2023-07-14T17:42:47.542183+0000","last_fullsized":"2023-07-14T17:42:47.542183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:21.155958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.762568+0000","last_change":"2023-07-14T17:42:41.917527+0000","last_active":"2023-07-14T17:43:02.762568+0000","last_peered":"2023-07-14T17:43:02.762568+0000","last_clean":"2023-07-14T17:43:02.762568+0000","last_became_active":"2023-07-14T17:42:41.917239+0000","last_became_peered":"2023-07-14T17:42:41.917239+0000","last_unstale":"2023-07-14T17:43:02.762568+0000","last_undegraded":"2023-07-14T17:43:02.762568+0000","last_fullsized":"2023-07-14T17:43:02.762568+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:04:18.619214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.315351+0000","last_change":"2023-07-14T17:42:42.554866+0000","last_active":"2023-07-14T17:42:48.315351+0000","last_peered":"2023-07-14T17:42:48.315351+0000","last_clean":"2023-07-14T17:42:48.315351+0000","last_became_active":"2023-07-14T17:42:42.554585+0000","last_became_peered":"2023-07-14T17:42:42.554585+0000","last_unstale":"2023-07-14T17:42:48.315351+0000","last_undegraded":"2023-07-14T17:42:48.315351+0000","last_fullsized":"2023-07-14T17:42:48.315351+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:01.732200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833167+0000","last_change":"2023-07-14T17:42:46.544788+0000","last_active":"2023-07-14T17:42:47.833167+0000","last_peered":"2023-07-14T17:42:47.833167+0000","last_clean":"2023-07-14T17:42:47.833167+0000","last_became_active":"2023-07-14T17:42:46.544419+0000","last_became_peered":"2023-07-14T17:42:46.544419+0000","last_unstale":"2023-07-14T17:42:47.833167+0000","last_undegraded":"2023-07-14T17:42:47.833167+0000","last_fullsized":"2023-07-14T17:42:47.833167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:03:38.940614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"86'24","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.785099+0000","last_change":"2023-07-14T17:42:41.917069+0000","last_active":"2023-07-14T17:43:04.785099+0000","last_peered":"2023-07-14T17:43:04.785099+0000","last_clean":"2023-07-14T17:43:04.785099+0000","last_became_active":"2023-07-14T17:42:41.916630+0000","last_became_peered":"2023-07-14T17:42:41.916630+0000","last_unstale":"2023-07-14T17:43:04.785099+0000","last_undegraded":"2023-07-14T17:43:04.785099+0000","last_fullsized":"2023-07-14T17:43:04.785099+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:05:57.979299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38515,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":39,"num_write":26,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835769+0000","last_change":"2023-07-14T17:42:42.542124+0000","last_active":"2023-07-14T17:42:47.835769+0000","last_peered":"2023-07-14T17:42:47.835769+0000","last_clean":"2023-07-14T17:42:47.835769+0000","last_became_active":"2023-07-14T17:42:42.541851+0000","last_became_peered":"2023-07-14T17:42:42.541851+0000","last_unstale":"2023-07-14T17:42:47.835769+0000","last_undegraded":"2023-07-14T17:42:47.835769+0000","last_fullsized":"2023-07-14T17:42:47.835769+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:06.253283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540680+0000","last_change":"2023-07-14T17:42:46.546167+0000","last_active":"2023-07-14T17:42:47.540680+0000","last_peered":"2023-07-14T17:42:47.540680+0000","last_clean":"2023-07-14T17:42:47.540680+0000","last_became_active":"2023-07-14T17:42:46.545768+0000","last_became_peered":"2023-07-14T17:42:46.545768+0000","last_unstale":"2023-07-14T17:42:47.540680+0000","last_undegraded":"2023-07-14T17:42:47.540680+0000","last_fullsized":"2023-07-14T17:42:47.540680+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:35:53.682315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.781459+0000","last_change":"2023-07-14T17:42:41.918858+0000","last_active":"2023-07-14T17:43:04.781459+0000","last_peered":"2023-07-14T17:43:04.781459+0000","last_clean":"2023-07-14T17:43:04.781459+0000","last_became_active":"2023-07-14T17:42:41.918568+0000","last_became_peered":"2023-07-14T17:42:41.918568+0000","last_unstale":"2023-07-14T17:43:04.781459+0000","last_undegraded":"2023-07-14T17:43:04.781459+0000","last_fullsized":"2023-07-14T17:43:04.781459+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:14:24.080564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313819+0000","last_change":"2023-07-14T17:42:42.542310+0000","last_active":"2023-07-14T17:42:48.313819+0000","last_peered":"2023-07-14T17:42:48.313819+0000","last_clean":"2023-07-14T17:42:48.313819+0000","last_became_active":"2023-07-14T17:42:42.541989+0000","last_became_peered":"2023-07-14T17:42:42.541989+0000","last_unstale":"2023-07-14T17:42:48.313819+0000","last_undegraded":"2023-07-14T17:42:48.313819+0000","last_fullsized":"2023-07-14T17:42:48.313819+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:41:45.629010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059652+0000","last_change":"2023-07-14T17:42:46.548737+0000","last_active":"2023-07-14T17:42:48.059652+0000","last_peered":"2023-07-14T17:42:48.059652+0000","last_clean":"2023-07-14T17:42:48.059652+0000","last_became_active":"2023-07-14T17:42:46.548255+0000","last_became_peered":"2023-07-14T17:42:46.548255+0000","last_unstale":"2023-07-14T17:42:48.059652+0000","last_undegraded":"2023-07-14T17:42:48.059652+0000","last_fullsized":"2023-07-14T17:42:48.059652+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:10.311183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.770445+0000","last_change":"2023-07-14T17:42:41.916382+0000","last_active":"2023-07-14T17:43:03.770445+0000","last_peered":"2023-07-14T17:43:03.770445+0000","last_clean":"2023-07-14T17:43:03.770445+0000","last_became_active":"2023-07-14T17:42:41.916082+0000","last_became_peered":"2023-07-14T17:42:41.916082+0000","last_unstale":"2023-07-14T17:43:03.770445+0000","last_undegraded":"2023-07-14T17:43:03.770445+0000","last_fullsized":"2023-07-14T17:43:03.770445+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:54:15.440758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313921+0000","last_change":"2023-07-14T17:42:42.546812+0000","last_active":"2023-07-14T17:42:48.313921+0000","last_peered":"2023-07-14T17:42:48.313921+0000","last_clean":"2023-07-14T17:42:48.313921+0000","last_became_active":"2023-07-14T17:42:42.546517+0000","last_became_peered":"2023-07-14T17:42:42.546517+0000","last_unstale":"2023-07-14T17:42:48.313921+0000","last_undegraded":"2023-07-14T17:42:48.313921+0000","last_fullsized":"2023-07-14T17:42:48.313921+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:41:22.699813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833687+0000","last_change":"2023-07-14T17:42:46.547726+0000","last_active":"2023-07-14T17:42:47.833687+0000","last_peered":"2023-07-14T17:42:47.833687+0000","last_clean":"2023-07-14T17:42:47.833687+0000","last_became_active":"2023-07-14T17:42:46.547207+0000","last_became_peered":"2023-07-14T17:42:46.547207+0000","last_unstale":"2023-07-14T17:42:47.833687+0000","last_undegraded":"2023-07-14T17:42:47.833687+0000","last_fullsized":"2023-07-14T17:42:47.833687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:55:16.418869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.761166+0000","last_change":"2023-07-14T17:42:41.913802+0000","last_active":"2023-07-14T17:43:02.761166+0000","last_peered":"2023-07-14T17:43:02.761166+0000","last_clean":"2023-07-14T17:43:02.761166+0000","last_became_active":"2023-07-14T17:42:41.913467+0000","last_became_peered":"2023-07-14T17:42:41.913467+0000","last_unstale":"2023-07-14T17:43:02.761166+0000","last_undegraded":"2023-07-14T17:43:02.761166+0000","last_fullsized":"2023-07-14T17:43:02.761166+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:46:59.725226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313504+0000","last_change":"2023-07-14T17:42:42.545379+0000","last_active":"2023-07-14T17:42:48.313504+0000","last_peered":"2023-07-14T17:42:48.313504+0000","last_clean":"2023-07-14T17:42:48.313504+0000","last_became_active":"2023-07-14T17:42:42.545012+0000","last_became_peered":"2023-07-14T17:42:42.545012+0000","last_unstale":"2023-07-14T17:42:48.313504+0000","last_undegraded":"2023-07-14T17:42:48.313504+0000","last_fullsized":"2023-07-14T17:42:48.313504+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:10.885034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313517+0000","last_change":"2023-07-14T17:42:46.544524+0000","last_active":"2023-07-14T17:42:48.313517+0000","last_peered":"2023-07-14T17:42:48.313517+0000","last_clean":"2023-07-14T17:42:48.313517+0000","last_became_active":"2023-07-14T17:42:46.544152+0000","last_became_peered":"2023-07-14T17:42:46.544152+0000","last_unstale":"2023-07-14T17:42:48.313517+0000","last_undegraded":"2023-07-14T17:42:48.313517+0000","last_fullsized":"2023-07-14T17:42:48.313517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:01:14.398239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'16","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:01.756350+0000","last_change":"2023-07-14T17:42:41.917282+0000","last_active":"2023-07-14T17:43:01.756350+0000","last_peered":"2023-07-14T17:43:01.756350+0000","last_clean":"2023-07-14T17:43:01.756350+0000","last_became_active":"2023-07-14T17:42:41.916994+0000","last_became_peered":"2023-07-14T17:42:41.916994+0000","last_unstale":"2023-07-14T17:43:01.756350+0000","last_undegraded":"2023-07-14T17:43:01.756350+0000","last_fullsized":"2023-07-14T17:43:01.756350+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:41:27.420237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540359+0000","last_change":"2023-07-14T17:42:42.548525+0000","last_active":"2023-07-14T17:42:47.540359+0000","last_peered":"2023-07-14T17:42:47.540359+0000","last_clean":"2023-07-14T17:42:47.540359+0000","last_became_active":"2023-07-14T17:42:42.548270+0000","last_became_peered":"2023-07-14T17:42:42.548270+0000","last_unstale":"2023-07-14T17:42:47.540359+0000","last_undegraded":"2023-07-14T17:42:47.540359+0000","last_fullsized":"2023-07-14T17:42:47.540359+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:18:32.383297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540365+0000","last_change":"2023-07-14T17:42:46.546480+0000","last_active":"2023-07-14T17:42:47.540365+0000","last_peered":"2023-07-14T17:42:47.540365+0000","last_clean":"2023-07-14T17:42:47.540365+0000","last_became_active":"2023-07-14T17:42:46.546078+0000","last_became_peered":"2023-07-14T17:42:46.546078+0000","last_unstale":"2023-07-14T17:42:47.540365+0000","last_undegraded":"2023-07-14T17:42:47.540365+0000","last_fullsized":"2023-07-14T17:42:47.540365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:17:38.547810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835965+0000","last_change":"2023-07-14T17:42:41.914226+0000","last_active":"2023-07-14T17:42:47.835965+0000","last_peered":"2023-07-14T17:42:47.835965+0000","last_clean":"2023-07-14T17:42:47.835965+0000","last_became_active":"2023-07-14T17:42:41.913820+0000","last_became_peered":"2023-07-14T17:42:41.913820+0000","last_unstale":"2023-07-14T17:42:47.835965+0000","last_undegraded":"2023-07-14T17:42:47.835965+0000","last_fullsized":"2023-07-14T17:42:47.835965+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:31:36.040725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059864+0000","last_change":"2023-07-14T17:42:42.543549+0000","last_active":"2023-07-14T17:42:48.059864+0000","last_peered":"2023-07-14T17:42:48.059864+0000","last_clean":"2023-07-14T17:42:48.059864+0000","last_became_active":"2023-07-14T17:42:42.543245+0000","last_became_peered":"2023-07-14T17:42:42.543245+0000","last_unstale":"2023-07-14T17:42:48.059864+0000","last_undegraded":"2023-07-14T17:42:48.059864+0000","last_fullsized":"2023-07-14T17:42:48.059864+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:30:07.889057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835441+0000","last_change":"2023-07-14T17:42:46.577245+0000","last_active":"2023-07-14T17:42:47.835441+0000","last_peered":"2023-07-14T17:42:47.835441+0000","last_clean":"2023-07-14T17:42:47.835441+0000","last_became_active":"2023-07-14T17:42:46.576793+0000","last_became_peered":"2023-07-14T17:42:46.576793+0000","last_unstale":"2023-07-14T17:42:47.835441+0000","last_undegraded":"2023-07-14T17:42:47.835441+0000","last_fullsized":"2023-07-14T17:42:47.835441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:29:32.688546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833389+0000","last_change":"2023-07-14T17:42:41.917199+0000","last_active":"2023-07-14T17:42:47.833389+0000","last_peered":"2023-07-14T17:42:47.833389+0000","last_clean":"2023-07-14T17:42:47.833389+0000","last_became_active":"2023-07-14T17:42:41.916924+0000","last_became_peered":"2023-07-14T17:42:41.916924+0000","last_unstale":"2023-07-14T17:42:47.833389+0000","last_undegraded":"2023-07-14T17:42:47.833389+0000","last_fullsized":"2023-07-14T17:42:47.833389+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:12:02.515835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314451+0000","last_change":"2023-07-14T17:42:42.540531+0000","last_active":"2023-07-14T17:42:48.314451+0000","last_peered":"2023-07-14T17:42:48.314451+0000","last_clean":"2023-07-14T17:42:48.314451+0000","last_became_active":"2023-07-14T17:42:42.540165+0000","last_became_peered":"2023-07-14T17:42:42.540165+0000","last_unstale":"2023-07-14T17:42:48.314451+0000","last_undegraded":"2023-07-14T17:42:48.314451+0000","last_fullsized":"2023-07-14T17:42:48.314451+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:34:21.411165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313334+0000","last_change":"2023-07-14T17:42:46.537158+0000","last_active":"2023-07-14T17:42:48.313334+0000","last_peered":"2023-07-14T17:42:48.313334+0000","last_clean":"2023-07-14T17:42:48.313334+0000","last_became_active":"2023-07-14T17:42:46.536816+0000","last_became_peered":"2023-07-14T17:42:46.536816+0000","last_unstale":"2023-07-14T17:42:48.313334+0000","last_undegraded":"2023-07-14T17:42:48.313334+0000","last_fullsized":"2023-07-14T17:42:48.313334+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:47:53.565328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313911+0000","last_change":"2023-07-14T17:42:41.913638+0000","last_active":"2023-07-14T17:42:48.313911+0000","last_peered":"2023-07-14T17:42:48.313911+0000","last_clean":"2023-07-14T17:42:48.313911+0000","last_became_active":"2023-07-14T17:42:41.913361+0000","last_became_peered":"2023-07-14T17:42:41.913361+0000","last_unstale":"2023-07-14T17:42:48.313911+0000","last_undegraded":"2023-07-14T17:42:48.313911+0000","last_fullsized":"2023-07-14T17:42:48.313911+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:30:25.777267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314326+0000","last_change":"2023-07-14T17:42:42.541738+0000","last_active":"2023-07-14T17:42:48.314326+0000","last_peered":"2023-07-14T17:42:48.314326+0000","last_clean":"2023-07-14T17:42:48.314326+0000","last_became_active":"2023-07-14T17:42:42.541264+0000","last_became_peered":"2023-07-14T17:42:42.541264+0000","last_unstale":"2023-07-14T17:42:48.314326+0000","last_undegraded":"2023-07-14T17:42:48.314326+0000","last_fullsized":"2023-07-14T17:42:48.314326+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:38:28.774135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313008+0000","last_change":"2023-07-14T17:42:46.576999+0000","last_active":"2023-07-14T17:42:48.313008+0000","last_peered":"2023-07-14T17:42:48.313008+0000","last_clean":"2023-07-14T17:42:48.313008+0000","last_became_active":"2023-07-14T17:42:46.576652+0000","last_became_peered":"2023-07-14T17:42:46.576652+0000","last_unstale":"2023-07-14T17:42:48.313008+0000","last_undegraded":"2023-07-14T17:42:48.313008+0000","last_fullsized":"2023-07-14T17:42:48.313008+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:59:23.913753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540039+0000","last_change":"2023-07-14T17:42:42.547053+0000","last_active":"2023-07-14T17:42:47.540039+0000","last_peered":"2023-07-14T17:42:47.540039+0000","last_clean":"2023-07-14T17:42:47.540039+0000","last_became_active":"2023-07-14T17:42:42.546735+0000","last_became_peered":"2023-07-14T17:42:42.546735+0000","last_unstale":"2023-07-14T17:42:47.540039+0000","last_undegraded":"2023-07-14T17:42:47.540039+0000","last_fullsized":"2023-07-14T17:42:47.540039+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:19:26.849479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.781254+0000","last_change":"2023-07-14T17:42:41.918692+0000","last_active":"2023-07-14T17:43:04.781254+0000","last_peered":"2023-07-14T17:43:04.781254+0000","last_clean":"2023-07-14T17:43:04.781254+0000","last_became_active":"2023-07-14T17:42:41.918360+0000","last_became_peered":"2023-07-14T17:42:41.918360+0000","last_unstale":"2023-07-14T17:43:04.781254+0000","last_undegraded":"2023-07-14T17:43:04.781254+0000","last_fullsized":"2023-07-14T17:43:04.781254+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:40:45.927843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314217+0000","last_change":"2023-07-14T17:42:46.543997+0000","last_active":"2023-07-14T17:42:48.314217+0000","last_peered":"2023-07-14T17:42:48.314217+0000","last_clean":"2023-07-14T17:42:48.314217+0000","last_became_active":"2023-07-14T17:42:46.543650+0000","last_became_peered":"2023-07-14T17:42:46.543650+0000","last_unstale":"2023-07-14T17:42:48.314217+0000","last_undegraded":"2023-07-14T17:42:48.314217+0000","last_fullsized":"2023-07-14T17:42:48.314217+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:25:35.518988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539400+0000","last_change":"2023-07-14T17:42:42.548860+0000","last_active":"2023-07-14T17:42:47.539400+0000","last_peered":"2023-07-14T17:42:47.539400+0000","last_clean":"2023-07-14T17:42:47.539400+0000","last_became_active":"2023-07-14T17:42:42.548564+0000","last_became_peered":"2023-07-14T17:42:42.548564+0000","last_unstale":"2023-07-14T17:42:47.539400+0000","last_undegraded":"2023-07-14T17:42:47.539400+0000","last_fullsized":"2023-07-14T17:42:47.539400+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:07:45.387117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'5","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.761783+0000","last_change":"2023-07-14T17:42:41.912601+0000","last_active":"2023-07-14T17:43:02.761783+0000","last_peered":"2023-07-14T17:43:02.761783+0000","last_clean":"2023-07-14T17:43:02.761783+0000","last_became_active":"2023-07-14T17:42:41.912265+0000","last_became_peered":"2023-07-14T17:42:41.912265+0000","last_unstale":"2023-07-14T17:43:02.761783+0000","last_undegraded":"2023-07-14T17:43:02.761783+0000","last_fullsized":"2023-07-14T17:43:02.761783+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:58.872606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":343,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.312950+0000","last_change":"2023-07-14T17:41:43.696093+0000","last_active":"2023-07-14T17:42:48.312950+0000","last_peered":"2023-07-14T17:42:48.312950+0000","last_clean":"2023-07-14T17:42:48.312950+0000","last_became_active":"2023-07-14T17:41:43.695511+0000","last_became_peered":"2023-07-14T17:41:43.695511+0000","last_unstale":"2023-07-14T17:42:48.312950+0000","last_undegraded":"2023-07-14T17:42:48.312950+0000","last_fullsized":"2023-07-14T17:42:48.312950+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:38:50.657865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:38:50.657865+0000","last_clean_scrub_stamp":"2023-07-14T17:38:50.657865+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:30:55.242715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540596+0000","last_change":"2023-07-14T17:42:46.577540+0000","last_active":"2023-07-14T17:42:47.540596+0000","last_peered":"2023-07-14T17:42:47.540596+0000","last_clean":"2023-07-14T17:42:47.540596+0000","last_became_active":"2023-07-14T17:42:46.577067+0000","last_became_peered":"2023-07-14T17:42:46.577067+0000","last_unstale":"2023-07-14T17:42:47.540596+0000","last_undegraded":"2023-07-14T17:42:47.540596+0000","last_fullsized":"2023-07-14T17:42:47.540596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:39:25.538930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313453+0000","last_change":"2023-07-14T17:42:42.543243+0000","last_active":"2023-07-14T17:42:48.313453+0000","last_peered":"2023-07-14T17:42:48.313453+0000","last_clean":"2023-07-14T17:42:48.313453+0000","last_became_active":"2023-07-14T17:42:42.542965+0000","last_became_peered":"2023-07-14T17:42:42.542965+0000","last_unstale":"2023-07-14T17:42:48.313453+0000","last_undegraded":"2023-07-14T17:42:48.313453+0000","last_fullsized":"2023-07-14T17:42:48.313453+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:08:21.008138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313410+0000","last_change":"2023-07-14T17:42:41.920806+0000","last_active":"2023-07-14T17:42:48.313410+0000","last_peered":"2023-07-14T17:42:48.313410+0000","last_clean":"2023-07-14T17:42:48.313410+0000","last_became_active":"2023-07-14T17:42:41.920528+0000","last_became_peered":"2023-07-14T17:42:41.920528+0000","last_unstale":"2023-07-14T17:42:48.313410+0000","last_undegraded":"2023-07-14T17:42:48.313410+0000","last_fullsized":"2023-07-14T17:42:48.313410+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:43:43.408841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059657+0000","last_change":"2023-07-14T17:42:46.542026+0000","last_active":"2023-07-14T17:42:48.059657+0000","last_peered":"2023-07-14T17:42:48.059657+0000","last_clean":"2023-07-14T17:42:48.059657+0000","last_became_active":"2023-07-14T17:42:46.541672+0000","last_became_peered":"2023-07-14T17:42:46.541672+0000","last_unstale":"2023-07-14T17:42:48.059657+0000","last_undegraded":"2023-07-14T17:42:48.059657+0000","last_fullsized":"2023-07-14T17:42:48.059657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:45:35.131993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"86'3","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.761841+0000","last_change":"2023-07-14T17:42:41.915798+0000","last_active":"2023-07-14T17:43:02.761841+0000","last_peered":"2023-07-14T17:43:02.761841+0000","last_clean":"2023-07-14T17:43:02.761841+0000","last_became_active":"2023-07-14T17:42:41.915492+0000","last_became_peered":"2023-07-14T17:42:41.915492+0000","last_unstale":"2023-07-14T17:43:02.761841+0000","last_undegraded":"2023-07-14T17:43:02.761841+0000","last_fullsized":"2023-07-14T17:43:02.761841+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:25:42.031839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313564+0000","last_change":"2023-07-14T17:42:42.533500+0000","last_active":"2023-07-14T17:42:48.313564+0000","last_peered":"2023-07-14T17:42:48.313564+0000","last_clean":"2023-07-14T17:42:48.313564+0000","last_became_active":"2023-07-14T17:42:42.533232+0000","last_became_peered":"2023-07-14T17:42:42.533232+0000","last_unstale":"2023-07-14T17:42:48.313564+0000","last_undegraded":"2023-07-14T17:42:48.313564+0000","last_fullsized":"2023-07-14T17:42:48.313564+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:57:14.212427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539366+0000","last_change":"2023-07-14T17:42:46.577131+0000","last_active":"2023-07-14T17:42:47.539366+0000","last_peered":"2023-07-14T17:42:47.539366+0000","last_clean":"2023-07-14T17:42:47.539366+0000","last_became_active":"2023-07-14T17:42:46.576695+0000","last_became_peered":"2023-07-14T17:42:46.576695+0000","last_unstale":"2023-07-14T17:42:47.539366+0000","last_undegraded":"2023-07-14T17:42:47.539366+0000","last_fullsized":"2023-07-14T17:42:47.539366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:49:05.554462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'7","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.762829+0000","last_change":"2023-07-14T17:42:41.919021+0000","last_active":"2023-07-14T17:43:02.762829+0000","last_peered":"2023-07-14T17:43:02.762829+0000","last_clean":"2023-07-14T17:43:02.762829+0000","last_became_active":"2023-07-14T17:42:41.918714+0000","last_became_peered":"2023-07-14T17:42:41.918714+0000","last_unstale":"2023-07-14T17:43:02.762829+0000","last_undegraded":"2023-07-14T17:43:02.762829+0000","last_fullsized":"2023-07-14T17:43:02.762829+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:40:48.400679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539836+0000","last_change":"2023-07-14T17:42:42.534661+0000","last_active":"2023-07-14T17:42:47.539836+0000","last_peered":"2023-07-14T17:42:47.539836+0000","last_clean":"2023-07-14T17:42:47.539836+0000","last_became_active":"2023-07-14T17:42:42.534357+0000","last_became_peered":"2023-07-14T17:42:42.534357+0000","last_unstale":"2023-07-14T17:42:47.539836+0000","last_undegraded":"2023-07-14T17:42:47.539836+0000","last_fullsized":"2023-07-14T17:42:47.539836+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:24:53.645011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.122946+0000","last_change":"2023-07-14T17:42:46.538648+0000","last_active":"2023-07-14T17:42:48.122946+0000","last_peered":"2023-07-14T17:42:48.122946+0000","last_clean":"2023-07-14T17:42:48.122946+0000","last_became_active":"2023-07-14T17:42:46.538225+0000","last_became_peered":"2023-07-14T17:42:46.538225+0000","last_unstale":"2023-07-14T17:42:48.122946+0000","last_undegraded":"2023-07-14T17:42:48.122946+0000","last_fullsized":"2023-07-14T17:42:48.122946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:36:57.320814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'5","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:00.743421+0000","last_change":"2023-07-14T17:42:41.915406+0000","last_active":"2023-07-14T17:43:00.743421+0000","last_peered":"2023-07-14T17:43:00.743421+0000","last_clean":"2023-07-14T17:43:00.743421+0000","last_became_active":"2023-07-14T17:42:41.914967+0000","last_became_peered":"2023-07-14T17:42:41.914967+0000","last_unstale":"2023-07-14T17:43:00.743421+0000","last_undegraded":"2023-07-14T17:43:00.743421+0000","last_fullsized":"2023-07-14T17:43:00.743421+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:56:07.113627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313341+0000","last_change":"2023-07-14T17:42:42.539052+0000","last_active":"2023-07-14T17:42:48.313341+0000","last_peered":"2023-07-14T17:42:48.313341+0000","last_clean":"2023-07-14T17:42:48.313341+0000","last_became_active":"2023-07-14T17:42:42.538752+0000","last_became_peered":"2023-07-14T17:42:42.538752+0000","last_unstale":"2023-07-14T17:42:48.313341+0000","last_undegraded":"2023-07-14T17:42:48.313341+0000","last_fullsized":"2023-07-14T17:42:48.313341+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:25:31.544906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542387+0000","last_change":"2023-07-14T17:42:46.577224+0000","last_active":"2023-07-14T17:42:47.542387+0000","last_peered":"2023-07-14T17:42:47.542387+0000","last_clean":"2023-07-14T17:42:47.542387+0000","last_became_active":"2023-07-14T17:42:46.576781+0000","last_became_peered":"2023-07-14T17:42:46.576781+0000","last_unstale":"2023-07-14T17:42:47.542387+0000","last_undegraded":"2023-07-14T17:42:47.542387+0000","last_fullsized":"2023-07-14T17:42:47.542387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:47:23.639743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:00.744812+0000","last_change":"2023-07-14T17:42:41.916406+0000","last_active":"2023-07-14T17:43:00.744812+0000","last_peered":"2023-07-14T17:43:00.744812+0000","last_clean":"2023-07-14T17:43:00.744812+0000","last_became_active":"2023-07-14T17:42:41.916115+0000","last_became_peered":"2023-07-14T17:42:41.916115+0000","last_unstale":"2023-07-14T17:43:00.744812+0000","last_undegraded":"2023-07-14T17:43:00.744812+0000","last_fullsized":"2023-07-14T17:43:00.744812+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:59:35.868770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835280+0000","last_change":"2023-07-14T17:42:42.529969+0000","last_active":"2023-07-14T17:42:47.835280+0000","last_peered":"2023-07-14T17:42:47.835280+0000","last_clean":"2023-07-14T17:42:47.835280+0000","last_became_active":"2023-07-14T17:42:42.529659+0000","last_became_peered":"2023-07-14T17:42:42.529659+0000","last_unstale":"2023-07-14T17:42:47.835280+0000","last_undegraded":"2023-07-14T17:42:47.835280+0000","last_fullsized":"2023-07-14T17:42:47.835280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:45:58.093667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833742+0000","last_change":"2023-07-14T17:42:46.546303+0000","last_active":"2023-07-14T17:42:47.833742+0000","last_peered":"2023-07-14T17:42:47.833742+0000","last_clean":"2023-07-14T17:42:47.833742+0000","last_became_active":"2023-07-14T17:42:46.545947+0000","last_became_peered":"2023-07-14T17:42:46.545947+0000","last_unstale":"2023-07-14T17:42:47.833742+0000","last_undegraded":"2023-07-14T17:42:47.833742+0000","last_fullsized":"2023-07-14T17:42:47.833742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:08:49.113221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"86'5","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.761458+0000","last_change":"2023-07-14T17:42:41.916373+0000","last_active":"2023-07-14T17:43:02.761458+0000","last_peered":"2023-07-14T17:43:02.761458+0000","last_clean":"2023-07-14T17:43:02.761458+0000","last_became_active":"2023-07-14T17:42:41.915966+0000","last_became_peered":"2023-07-14T17:42:41.915966+0000","last_unstale":"2023-07-14T17:43:02.761458+0000","last_undegraded":"2023-07-14T17:43:02.761458+0000","last_fullsized":"2023-07-14T17:43:02.761458+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:40:26.456139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059423+0000","last_change":"2023-07-14T17:42:42.532040+0000","last_active":"2023-07-14T17:42:48.059423+0000","last_peered":"2023-07-14T17:42:48.059423+0000","last_clean":"2023-07-14T17:42:48.059423+0000","last_became_active":"2023-07-14T17:42:42.531690+0000","last_became_peered":"2023-07-14T17:42:42.531690+0000","last_unstale":"2023-07-14T17:42:48.059423+0000","last_undegraded":"2023-07-14T17:42:48.059423+0000","last_fullsized":"2023-07-14T17:42:48.059423+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:36:37.658897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542001+0000","last_change":"2023-07-14T17:42:46.550166+0000","last_active":"2023-07-14T17:42:47.542001+0000","last_peered":"2023-07-14T17:42:47.542001+0000","last_clean":"2023-07-14T17:42:47.542001+0000","last_became_active":"2023-07-14T17:42:46.549700+0000","last_became_peered":"2023-07-14T17:42:46.549700+0000","last_unstale":"2023-07-14T17:42:47.542001+0000","last_undegraded":"2023-07-14T17:42:47.542001+0000","last_fullsized":"2023-07-14T17:42:47.542001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:23.561271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"86'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.762988+0000","last_change":"2023-07-14T17:42:41.917309+0000","last_active":"2023-07-14T17:43:02.762988+0000","last_peered":"2023-07-14T17:43:02.762988+0000","last_clean":"2023-07-14T17:43:02.762988+0000","last_became_active":"2023-07-14T17:42:41.916694+0000","last_became_peered":"2023-07-14T17:42:41.916694+0000","last_unstale":"2023-07-14T17:43:02.762988+0000","last_undegraded":"2023-07-14T17:43:02.762988+0000","last_fullsized":"2023-07-14T17:43:02.762988+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:12:41.209743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539969+0000","last_change":"2023-07-14T17:42:42.531789+0000","last_active":"2023-07-14T17:42:47.539969+0000","last_peered":"2023-07-14T17:42:47.539969+0000","last_clean":"2023-07-14T17:42:47.539969+0000","last_became_active":"2023-07-14T17:42:42.531460+0000","last_became_peered":"2023-07-14T17:42:42.531460+0000","last_unstale":"2023-07-14T17:42:47.539969+0000","last_undegraded":"2023-07-14T17:42:47.539969+0000","last_fullsized":"2023-07-14T17:42:47.539969+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:17.645748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.541998+0000","last_change":"2023-07-14T17:42:46.577034+0000","last_active":"2023-07-14T17:42:47.541998+0000","last_peered":"2023-07-14T17:42:47.541998+0000","last_clean":"2023-07-14T17:42:47.541998+0000","last_became_active":"2023-07-14T17:42:46.576636+0000","last_became_peered":"2023-07-14T17:42:46.576636+0000","last_unstale":"2023-07-14T17:42:47.541998+0000","last_undegraded":"2023-07-14T17:42:47.541998+0000","last_fullsized":"2023-07-14T17:42:47.541998+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:08:40.397509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:00.743144+0000","last_change":"2023-07-14T17:42:41.919099+0000","last_active":"2023-07-14T17:43:00.743144+0000","last_peered":"2023-07-14T17:43:00.743144+0000","last_clean":"2023-07-14T17:43:00.743144+0000","last_became_active":"2023-07-14T17:42:41.918306+0000","last_became_peered":"2023-07-14T17:42:41.918306+0000","last_unstale":"2023-07-14T17:43:00.743144+0000","last_undegraded":"2023-07-14T17:43:00.743144+0000","last_fullsized":"2023-07-14T17:43:00.743144+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:20:09.398489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314097+0000","last_change":"2023-07-14T17:42:42.552969+0000","last_active":"2023-07-14T17:42:48.314097+0000","last_peered":"2023-07-14T17:42:48.314097+0000","last_clean":"2023-07-14T17:42:48.314097+0000","last_became_active":"2023-07-14T17:42:42.552647+0000","last_became_peered":"2023-07-14T17:42:42.552647+0000","last_unstale":"2023-07-14T17:42:48.314097+0000","last_undegraded":"2023-07-14T17:42:48.314097+0000","last_fullsized":"2023-07-14T17:42:48.314097+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:24:37.409124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314884+0000","last_change":"2023-07-14T17:42:46.538335+0000","last_active":"2023-07-14T17:42:48.314884+0000","last_peered":"2023-07-14T17:42:48.314884+0000","last_clean":"2023-07-14T17:42:48.314884+0000","last_became_active":"2023-07-14T17:42:46.537929+0000","last_became_peered":"2023-07-14T17:42:46.537929+0000","last_unstale":"2023-07-14T17:42:48.314884+0000","last_undegraded":"2023-07-14T17:42:48.314884+0000","last_fullsized":"2023-07-14T17:42:48.314884+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:42:43.435298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.785078+0000","last_change":"2023-07-14T17:42:41.916323+0000","last_active":"2023-07-14T17:43:04.785078+0000","last_peered":"2023-07-14T17:43:04.785078+0000","last_clean":"2023-07-14T17:43:04.785078+0000","last_became_active":"2023-07-14T17:42:41.916019+0000","last_became_peered":"2023-07-14T17:42:41.916019+0000","last_unstale":"2023-07-14T17:43:04.785078+0000","last_undegraded":"2023-07-14T17:43:04.785078+0000","last_fullsized":"2023-07-14T17:43:04.785078+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:42:06.067793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314443+0000","last_change":"2023-07-14T17:42:42.549843+0000","last_active":"2023-07-14T17:42:48.314443+0000","last_peered":"2023-07-14T17:42:48.314443+0000","last_clean":"2023-07-14T17:42:48.314443+0000","last_became_active":"2023-07-14T17:42:42.549347+0000","last_became_peered":"2023-07-14T17:42:42.549347+0000","last_unstale":"2023-07-14T17:42:48.314443+0000","last_undegraded":"2023-07-14T17:42:48.314443+0000","last_fullsized":"2023-07-14T17:42:48.314443+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:37:33.281681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313679+0000","last_change":"2023-07-14T17:42:46.545262+0000","last_active":"2023-07-14T17:42:48.313679+0000","last_peered":"2023-07-14T17:42:48.313679+0000","last_clean":"2023-07-14T17:42:48.313679+0000","last_became_active":"2023-07-14T17:42:46.544891+0000","last_became_peered":"2023-07-14T17:42:46.544891+0000","last_unstale":"2023-07-14T17:42:48.313679+0000","last_undegraded":"2023-07-14T17:42:48.313679+0000","last_fullsized":"2023-07-14T17:42:48.313679+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:58:10.135808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.770619+0000","last_change":"2023-07-14T17:42:41.912855+0000","last_active":"2023-07-14T17:43:03.770619+0000","last_peered":"2023-07-14T17:43:03.770619+0000","last_clean":"2023-07-14T17:43:03.770619+0000","last_became_active":"2023-07-14T17:42:41.912510+0000","last_became_peered":"2023-07-14T17:42:41.912510+0000","last_unstale":"2023-07-14T17:43:03.770619+0000","last_undegraded":"2023-07-14T17:43:03.770619+0000","last_fullsized":"2023-07-14T17:43:03.770619+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:58:29.251068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313890+0000","last_change":"2023-07-14T17:42:42.545344+0000","last_active":"2023-07-14T17:42:48.313890+0000","last_peered":"2023-07-14T17:42:48.313890+0000","last_clean":"2023-07-14T17:42:48.313890+0000","last_became_active":"2023-07-14T17:42:42.545056+0000","last_became_peered":"2023-07-14T17:42:42.545056+0000","last_unstale":"2023-07-14T17:42:48.313890+0000","last_undegraded":"2023-07-14T17:42:48.313890+0000","last_fullsized":"2023-07-14T17:42:48.313890+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:36:25.864767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539658+0000","last_change":"2023-07-14T17:42:46.541279+0000","last_active":"2023-07-14T17:42:47.539658+0000","last_peered":"2023-07-14T17:42:47.539658+0000","last_clean":"2023-07-14T17:42:47.539658+0000","last_became_active":"2023-07-14T17:42:46.540835+0000","last_became_peered":"2023-07-14T17:42:46.540835+0000","last_unstale":"2023-07-14T17:42:47.539658+0000","last_undegraded":"2023-07-14T17:42:47.539658+0000","last_fullsized":"2023-07-14T17:42:47.539658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:04:21.200711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.771177+0000","last_change":"2023-07-14T17:42:41.919257+0000","last_active":"2023-07-14T17:43:03.771177+0000","last_peered":"2023-07-14T17:43:03.771177+0000","last_clean":"2023-07-14T17:43:03.771177+0000","last_became_active":"2023-07-14T17:42:41.918968+0000","last_became_peered":"2023-07-14T17:42:41.918968+0000","last_unstale":"2023-07-14T17:43:03.771177+0000","last_undegraded":"2023-07-14T17:43:03.771177+0000","last_fullsized":"2023-07-14T17:43:03.771177+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:16:49.994602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313885+0000","last_change":"2023-07-14T17:42:42.543993+0000","last_active":"2023-07-14T17:42:48.313885+0000","last_peered":"2023-07-14T17:42:48.313885+0000","last_clean":"2023-07-14T17:42:48.313885+0000","last_became_active":"2023-07-14T17:42:42.543685+0000","last_became_peered":"2023-07-14T17:42:42.543685+0000","last_unstale":"2023-07-14T17:42:48.313885+0000","last_undegraded":"2023-07-14T17:42:48.313885+0000","last_fullsized":"2023-07-14T17:42:48.313885+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:17:07.258942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059529+0000","last_change":"2023-07-14T17:42:42.538553+0000","last_active":"2023-07-14T17:42:48.059529+0000","last_peered":"2023-07-14T17:42:48.059529+0000","last_clean":"2023-07-14T17:42:48.059529+0000","last_became_active":"2023-07-14T17:42:42.538251+0000","last_became_peered":"2023-07-14T17:42:42.538251+0000","last_unstale":"2023-07-14T17:42:48.059529+0000","last_undegraded":"2023-07-14T17:42:48.059529+0000","last_fullsized":"2023-07-14T17:42:48.059529+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:59:19.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123275+0000","last_change":"2023-07-14T17:42:42.543819+0000","last_active":"2023-07-14T17:42:48.123275+0000","last_peered":"2023-07-14T17:42:48.123275+0000","last_clean":"2023-07-14T17:42:48.123275+0000","last_became_active":"2023-07-14T17:42:42.543311+0000","last_became_peered":"2023-07-14T17:42:42.543311+0000","last_unstale":"2023-07-14T17:42:48.123275+0000","last_undegraded":"2023-07-14T17:42:48.123275+0000","last_fullsized":"2023-07-14T17:42:48.123275+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:43:18.425305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123144+0000","last_change":"2023-07-14T17:42:42.537525+0000","last_active":"2023-07-14T17:42:48.123144+0000","last_peered":"2023-07-14T17:42:48.123144+0000","last_clean":"2023-07-14T17:42:48.123144+0000","last_became_active":"2023-07-14T17:42:42.537242+0000","last_became_peered":"2023-07-14T17:42:42.537242+0000","last_unstale":"2023-07-14T17:42:48.123144+0000","last_undegraded":"2023-07-14T17:42:48.123144+0000","last_fullsized":"2023-07-14T17:42:48.123144+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:44:07.205354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313834+0000","last_change":"2023-07-14T17:42:42.547670+0000","last_active":"2023-07-14T17:42:48.313834+0000","last_peered":"2023-07-14T17:42:48.313834+0000","last_clean":"2023-07-14T17:42:48.313834+0000","last_became_active":"2023-07-14T17:42:42.547349+0000","last_became_peered":"2023-07-14T17:42:42.547349+0000","last_unstale":"2023-07-14T17:42:48.313834+0000","last_undegraded":"2023-07-14T17:42:48.313834+0000","last_fullsized":"2023-07-14T17:42:48.313834+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:11:40.630882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313964+0000","last_change":"2023-07-14T17:42:42.542741+0000","last_active":"2023-07-14T17:42:48.313964+0000","last_peered":"2023-07-14T17:42:48.313964+0000","last_clean":"2023-07-14T17:42:48.313964+0000","last_became_active":"2023-07-14T17:42:42.541972+0000","last_became_peered":"2023-07-14T17:42:42.541972+0000","last_unstale":"2023-07-14T17:42:48.313964+0000","last_undegraded":"2023-07-14T17:42:48.313964+0000","last_fullsized":"2023-07-14T17:42:48.313964+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:56:21.183121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059903+0000","last_change":"2023-07-14T17:42:42.535043+0000","last_active":"2023-07-14T17:42:48.059903+0000","last_peered":"2023-07-14T17:42:48.059903+0000","last_clean":"2023-07-14T17:42:48.059903+0000","last_became_active":"2023-07-14T17:42:42.534281+0000","last_became_peered":"2023-07-14T17:42:42.534281+0000","last_unstale":"2023-07-14T17:42:48.059903+0000","last_undegraded":"2023-07-14T17:42:48.059903+0000","last_fullsized":"2023-07-14T17:42:48.059903+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:59:12.865751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123808+0000","last_change":"2023-07-14T17:42:42.540866+0000","last_active":"2023-07-14T17:42:48.123808+0000","last_peered":"2023-07-14T17:42:48.123808+0000","last_clean":"2023-07-14T17:42:48.123808+0000","last_became_active":"2023-07-14T17:42:42.540568+0000","last_became_peered":"2023-07-14T17:42:42.540568+0000","last_unstale":"2023-07-14T17:42:48.123808+0000","last_undegraded":"2023-07-14T17:42:48.123808+0000","last_fullsized":"2023-07-14T17:42:48.123808+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:48:50.914640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540078+0000","last_change":"2023-07-14T17:42:42.541646+0000","last_active":"2023-07-14T17:42:47.540078+0000","last_peered":"2023-07-14T17:42:47.540078+0000","last_clean":"2023-07-14T17:42:47.540078+0000","last_became_active":"2023-07-14T17:42:42.541324+0000","last_became_peered":"2023-07-14T17:42:42.541324+0000","last_unstale":"2023-07-14T17:42:47.540078+0000","last_undegraded":"2023-07-14T17:42:47.540078+0000","last_fullsized":"2023-07-14T17:42:47.540078+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:11:55.891005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539881+0000","last_change":"2023-07-14T17:42:42.534650+0000","last_active":"2023-07-14T17:42:47.539881+0000","last_peered":"2023-07-14T17:42:47.539881+0000","last_clean":"2023-07-14T17:42:47.539881+0000","last_became_active":"2023-07-14T17:42:42.534351+0000","last_became_peered":"2023-07-14T17:42:42.534351+0000","last_unstale":"2023-07-14T17:42:47.539881+0000","last_undegraded":"2023-07-14T17:42:47.539881+0000","last_fullsized":"2023-07-14T17:42:47.539881+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:13:54.843952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539359+0000","last_change":"2023-07-14T17:42:42.535815+0000","last_active":"2023-07-14T17:42:47.539359+0000","last_peered":"2023-07-14T17:42:47.539359+0000","last_clean":"2023-07-14T17:42:47.539359+0000","last_became_active":"2023-07-14T17:42:42.535510+0000","last_became_peered":"2023-07-14T17:42:42.535510+0000","last_unstale":"2023-07-14T17:42:47.539359+0000","last_undegraded":"2023-07-14T17:42:47.539359+0000","last_fullsized":"2023-07-14T17:42:47.539359+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:39:09.947314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.315221+0000","last_change":"2023-07-14T17:42:42.551758+0000","last_active":"2023-07-14T17:42:48.315221+0000","last_peered":"2023-07-14T17:42:48.315221+0000","last_clean":"2023-07-14T17:42:48.315221+0000","last_became_active":"2023-07-14T17:42:42.551444+0000","last_became_peered":"2023-07-14T17:42:42.551444+0000","last_unstale":"2023-07-14T17:42:48.315221+0000","last_undegraded":"2023-07-14T17:42:48.315221+0000","last_fullsized":"2023-07-14T17:42:48.315221+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:42:02.798071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313340+0000","last_change":"2023-07-14T17:42:42.546355+0000","last_active":"2023-07-14T17:42:48.313340+0000","last_peered":"2023-07-14T17:42:48.313340+0000","last_clean":"2023-07-14T17:42:48.313340+0000","last_became_active":"2023-07-14T17:42:42.546079+0000","last_became_peered":"2023-07-14T17:42:42.546079+0000","last_unstale":"2023-07-14T17:42:48.313340+0000","last_undegraded":"2023-07-14T17:42:48.313340+0000","last_fullsized":"2023-07-14T17:42:48.313340+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:42:39.486035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059704+0000","last_change":"2023-07-14T17:42:42.543586+0000","last_active":"2023-07-14T17:42:48.059704+0000","last_peered":"2023-07-14T17:42:48.059704+0000","last_clean":"2023-07-14T17:42:48.059704+0000","last_became_active":"2023-07-14T17:42:42.543259+0000","last_became_peered":"2023-07-14T17:42:42.543259+0000","last_unstale":"2023-07-14T17:42:48.059704+0000","last_undegraded":"2023-07-14T17:42:48.059704+0000","last_fullsized":"2023-07-14T17:42:48.059704+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:24:35.934043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":43799,"num_objects":82,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":21,"num_read_kb":54,"num_write":188,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":132419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165,"ondisk_log_size":165,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481807,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":462514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]}]},{"osd":10,"up_from":69,"seq":296352743443,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28612,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716540,"statfs":{"total":95995035648,"available":95965736960,"internally_reserved":0,"allocated":1634304,"data_stored":1092893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.71699999999999997}]},{"osd":2,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.755},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.63600000000000001}]},{"osd":3,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.64600000000000002}]},{"osd":4,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.69099999999999995}]},{"osd":5,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.67800000000000005}]},{"osd":6,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.83099999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":8,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.873}]},{"osd":9,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.84599999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]}]},{"osd":9,"up_from":64,"seq":274877906967,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":462840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.29099999999999998}]},{"osd":1,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.375},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.42399999999999999}]},{"osd":2,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.45100000000000001}]},{"osd":3,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.311}]},{"osd":4,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.48999999999999999}]},{"osd":5,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.47399999999999998}]},{"osd":6,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.372},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.52500000000000002}]},{"osd":7,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.438}]},{"osd":8,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.53300000000000003}]},{"osd":10,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.46000000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":8,"up_from":57,"seq":244813135900,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":462670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53400000000000003,"15min":0.53300000000000003},"min":{"1min":0.35499999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.437}]},{"osd":1,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.372,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83899999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.40999999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.57999999999999996}]},{"osd":2,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45800000000000002,"15min":0.46100000000000002},"min":{"1min":0.27500000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60599999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.47199999999999998,"15min":0.47999999999999998},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56499999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.496}]},{"osd":3,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.36299999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.51900000000000002}]},{"osd":4,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85099999999999998,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64300000000000002,"15min":0.64200000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89700000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.66900000000000004}]},{"osd":5,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57399999999999995,"15min":0.58099999999999996},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72499999999999998,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.312,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.75,"5min":1.018,"15min":1.018},"last":0.89300000000000002}]},{"osd":6,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.66600000000000004,"15min":0.66800000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68500000000000005,"15min":0.69299999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84799999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.75600000000000001}]},{"osd":7,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.51000000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92400000000000004,"5min":1.008,"15min":1.008},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71099999999999997,"15min":0.70999999999999996},"min":{"1min":0.48899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88200000000000001,"5min":1.258,"15min":1.258},"last":0.82599999999999996}]},{"osd":9,"last update":"Fri Jul 14 17:42:19 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.76800000000000002}]},{"osd":10,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.80700000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]}]},{"osd":7,"up_from":49,"seq":210453397535,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28532,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716620,"statfs":{"total":95995035648,"available":95965818880,"internally_reserved":0,"allocated":1544192,"data_stored":1052882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.503},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.59499999999999997}]},{"osd":1,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.41499999999999998,"15min":0.42299999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.57799999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.46000000000000002,"15min":0.47099999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.51800000000000002,"5min":0.746,"15min":0.746},"last":0.68100000000000005}]},{"osd":2,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.628,"15min":0.60299999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":5.1050000000000004,"5min":5.1050000000000004,"15min":5.1050000000000004},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.71999999999999997,"5min":1.048,"15min":1.048},"last":0.83699999999999997}]},{"osd":3,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.436,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.59899999999999998,"15min":0.61199999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72499999999999998,"5min":0.995,"15min":0.995},"last":0.80600000000000005}]},{"osd":4,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.49299999999999999,"15min":0.499},"min":{"1min":0.312,"5min":0.31,"15min":0.31},"max":{"1min":0.59999999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.52000000000000002,"15min":0.53200000000000003},"min":{"1min":0.28899999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.66900000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.879}]},{"osd":5,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63400000000000001,"15min":0.63200000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64100000000000001,"15min":0.64300000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.94199999999999995,"5min":1.02,"15min":1.02},"last":0.96499999999999997}]},{"osd":6,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.64000000000000001,"15min":0.65000000000000002},"min":{"1min":0.435,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.94899999999999995}]},{"osd":8,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.45700000000000002,"5min":0.441,"15min":0.441},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.71299999999999997,"15min":0.72099999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0620000000000001,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":1.0620000000000001}]},{"osd":9,"last update":"Fri Jul 14 17:42:19 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":1.01},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.91200000000000003}]},{"osd":10,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":1.038},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.99099999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0249999999999999}]}]},{"osd":6,"up_from":44,"seq":188978561060,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":462606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:42:18 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.36199999999999999,"15min":0.35299999999999998},"min":{"1min":0.29799999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.33100000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.35099999999999998}]},{"osd":1,"last update":"Fri Jul 14 17:42:18 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.57899999999999996,"15min":0.56699999999999995},"min":{"1min":0.45400000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.58099999999999996}]},{"osd":2,"last update":"Fri Jul 14 17:42:18 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56999999999999995,"15min":0.56799999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.76700000000000002,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.438,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.53700000000000003}]},{"osd":3,"last update":"Fri Jul 14 17:42:18 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45600000000000002,"15min":0.45100000000000001},"min":{"1min":0.32200000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69299999999999995,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.46899999999999997,"15min":0.45300000000000001},"min":{"1min":0.34599999999999997,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.40200000000000002}]},{"osd":4,"last update":"Fri Jul 14 17:42:18 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.58999999999999997}]},{"osd":5,"last update":"Fri Jul 14 17:42:18 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.59999999999999998,"15min":0.59299999999999997},"min":{"1min":0.44400000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.66200000000000003,"15min":0.65500000000000003},"min":{"1min":0.45800000000000002,"5min":0.443,"15min":0.443},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.59899999999999998}]},{"osd":7,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.66600000000000004,"15min":0.66000000000000003},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.498},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.68999999999999995,"15min":0.68799999999999994},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.60799999999999998}]},{"osd":8,"last update":"Fri Jul 14 17:43:04 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.73599999999999999,"15min":0.755},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.98999999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.69499999999999995,"15min":0.71099999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92500000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.50600000000000001}]},{"osd":9,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.46200000000000002}]},{"osd":10,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.63300000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":484811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:42:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.48899999999999999,"15min":0.46200000000000002},"min":{"1min":0.38,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.82999999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.436},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.50600000000000001,"15min":0.46899999999999997},"min":{"1min":0.34499999999999997,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.40899999999999997}]},{"osd":2,"last update":"Fri Jul 14 17:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.54200000000000004,"15min":0.497},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.86099999999999999,"5min":0.879,"15min":0.879},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.52300000000000002,"15min":0.48899999999999999},"min":{"1min":0.36499999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.73299999999999998}]},{"osd":3,"last update":"Fri Jul 14 17:42:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.52900000000000003,"15min":0.51400000000000001},"min":{"1min":0.33400000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.81200000000000006,"5min":0.873,"15min":0.873},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.56499999999999995,"15min":0.54100000000000004},"min":{"1min":0.443,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.56299999999999994}]},{"osd":4,"last update":"Fri Jul 14 17:42:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.42799999999999999,"15min":0.42199999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66300000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.44,"15min":0.42799999999999999},"min":{"1min":0.29999999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.47199999999999998}]},{"osd":5,"last update":"Fri Jul 14 17:42:59 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.60899999999999999,"15min":0.58499999999999996},"min":{"1min":0.48299999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.58199999999999996,"15min":0.57099999999999995},"min":{"1min":0.42699999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.69699999999999995}]},{"osd":6,"last update":"Fri Jul 14 17:42:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.40899999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.41899999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82299999999999995,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.60099999999999998}]},{"osd":7,"last update":"Fri Jul 14 17:42:39 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.53000000000000003,"15min":0.51500000000000001},"min":{"1min":0.378,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86499999999999999,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52700000000000002,"15min":0.52400000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.64600000000000002}]},{"osd":8,"last update":"Fri Jul 14 17:42:59 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.65900000000000003,"15min":0.65600000000000003},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1,"5min":1,"15min":1},"last":0.747},{"interface":"front","average":{"1min":0.746,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.54000000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.81799999999999995}]},{"osd":9,"last update":"Fri Jul 14 17:42:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.625},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.68200000000000005}]},{"osd":10,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.78100000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":463950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 17:42:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.51700000000000002,"15min":0.505},"min":{"1min":0.503,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.54100000000000004,"15min":0.52100000000000002},"min":{"1min":0.45000000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.46200000000000002}]},{"osd":2,"last update":"Fri Jul 14 17:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55900000000000005,"15min":0.53400000000000003},"min":{"1min":0.40999999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56999999999999995,"15min":0.55300000000000005},"min":{"1min":0.379,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.441}]},{"osd":3,"last update":"Fri Jul 14 17:41:15 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44700000000000001,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.40999999999999998,"15min":0.40600000000000003},"min":{"1min":0.27900000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.61699999999999999,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.47699999999999998}]},{"osd":4,"last update":"Fri Jul 14 17:42:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.55400000000000005,"15min":0.54000000000000004},"min":{"1min":0.435,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.40699999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.85099999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.59799999999999998}]},{"osd":5,"last update":"Fri Jul 14 17:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56499999999999995,"15min":0.55100000000000005},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.755,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59599999999999997,"15min":0.58999999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84099999999999997,"5min":0.88,"15min":0.88},"last":0.499}]},{"osd":6,"last update":"Fri Jul 14 17:42:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.48599999999999999,"15min":0.47099999999999997},"min":{"1min":0.33200000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.501,"15min":0.48799999999999999},"min":{"1min":0.34300000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.53200000000000003}]},{"osd":7,"last update":"Fri Jul 14 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.66900000000000004}]},{"osd":8,"last update":"Fri Jul 14 17:42:02 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.58699999999999997}]},{"osd":9,"last update":"Fri Jul 14 17:42:19 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.64100000000000001}]},{"osd":10,"last update":"Fri Jul 14 17:42:44 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.624},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.63300000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":502440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.47499999999999998},"min":{"1min":0.41099999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.5},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.48299999999999998,"15min":0.46700000000000003},"min":{"1min":0.39100000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.53000000000000003}]},{"osd":1,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.53100000000000003,"15min":0.48299999999999998},"min":{"1min":0.35199999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.55300000000000005,"15min":0.50900000000000001},"min":{"1min":0.5,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.80300000000000005}]},{"osd":3,"last update":"Fri Jul 14 17:42:22 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.55700000000000005,"15min":0.54100000000000004},"min":{"1min":0.47099999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.54900000000000004,"15min":0.53600000000000003},"min":{"1min":0.436,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.59499999999999997}]},{"osd":4,"last update":"Fri Jul 14 17:41:35 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.51900000000000002,"15min":0.51300000000000001},"min":{"1min":0.38400000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.55400000000000005,"15min":0.54600000000000004},"min":{"1min":0.39900000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.86599999999999999}]},{"osd":5,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.45600000000000002,"15min":0.43099999999999999},"min":{"1min":0.30399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.45200000000000001,"15min":0.437},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56200000000000006}]},{"osd":6,"last update":"Fri Jul 14 17:42:22 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.58799999999999997,"15min":0.57599999999999996},"min":{"1min":0.48299999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.70999999999999996}]},{"osd":7,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.59099999999999997,"15min":0.57499999999999996},"min":{"1min":0.40200000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.66000000000000003,"15min":0.64600000000000002},"min":{"1min":0.52500000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.94499999999999995}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.61299999999999999,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.59299999999999997,"15min":0.57399999999999995},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.69399999999999995}]},{"osd":9,"last update":"Fri Jul 14 17:42:22 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.91800000000000004}]},{"osd":10,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.876},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.90600000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182449,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":462340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.33700000000000002,"15min":0.32100000000000001},"min":{"1min":0.28899999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.504,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.307},{"interface":"front","average":{"1min":0.437,"5min":0.35099999999999998,"15min":0.32900000000000001},"min":{"1min":0.29499999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.33300000000000002}]},{"osd":1,"last update":"Fri Jul 14 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.52600000000000002,"15min":0.5},"min":{"1min":0.41299999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.80600000000000005,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.53100000000000003,"15min":0.501},"min":{"1min":0.47099999999999997,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.84099999999999997,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.501}]},{"osd":2,"last update":"Fri Jul 14 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.54200000000000004,"15min":0.52200000000000002},"min":{"1min":0.42599999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.55800000000000005,"15min":0.52600000000000002},"min":{"1min":0.436,"5min":0.379,"15min":0.379},"max":{"1min":0.91900000000000004,"5min":1.423,"15min":1.423},"last":0.60999999999999999}]},{"osd":4,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58099999999999996,"15min":0.56699999999999995},"min":{"1min":0.37,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71299999999999997,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60299999999999998,"15min":0.58199999999999996},"min":{"1min":0.50600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84199999999999997,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.48999999999999999}]},{"osd":5,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.38100000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85299999999999998,"5min":1.593,"15min":1.593},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.45400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.85699999999999998,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.621}]},{"osd":6,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.53700000000000003,"15min":0.53300000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.78700000000000003,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.34999999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67500000000000004,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.42999999999999999}]},{"osd":7,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.63700000000000001,"15min":0.64300000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79300000000000004,"5min":1.28,"15min":1.28},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.65200000000000002,"15min":0.64400000000000002},"min":{"1min":0.56200000000000006,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.89000000000000001,"5min":1.03,"15min":1.03},"last":0.56999999999999995}]},{"osd":8,"last update":"Fri Jul 14 17:41:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.55800000000000005}]},{"osd":9,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.41699999999999998}]},{"osd":10,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.59999999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953516,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":462528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.35399999999999998,"5min":0.311,"15min":0.311},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.52100000000000002,"15min":0.502},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.496}]},{"osd":1,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.39700000000000002,"15min":0.375},"min":{"1min":0.30499999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.41099999999999998,"15min":0.39500000000000002},"min":{"1min":0.29799999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.67200000000000004,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.439}]},{"osd":2,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.42199999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.59399999999999997,"15min":0.57199999999999995},"min":{"1min":0.437,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85399999999999998,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.73399999999999999}]},{"osd":3,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.57899999999999996,"15min":0.54500000000000004},"min":{"1min":0.39600000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.96599999999999997,"5min":1.498,"15min":1.498},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55400000000000005,"15min":0.53200000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.84399999999999997,"5min":1.629,"15min":1.629},"last":0.57099999999999995}]},{"osd":5,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.63100000000000001,"15min":0.61099999999999999},"min":{"1min":0.51500000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89800000000000002,"5min":1.256,"15min":1.256},"last":0.755},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.624,"15min":0.60999999999999999},"min":{"1min":0.44900000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.68400000000000005}]},{"osd":6,"last update":"Fri Jul 14 17:42:18 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.745,"15min":0.753},"min":{"1min":0.51800000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91400000000000003,"5min":2.7789999999999999,"15min":2.7789999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.67900000000000005,"15min":0.66600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.2210000000000001,"5min":1.476,"15min":1.476},"last":0.55500000000000005}]},{"osd":7,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.55000000000000004,"15min":0.52300000000000002},"min":{"1min":0.39000000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.59399999999999997,"15min":0.57799999999999996},"min":{"1min":0.38100000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.63500000000000001}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.65600000000000003,"15min":0.64400000000000002},"min":{"1min":0.41999999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72299999999999998,"15min":0.71399999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.66100000000000003}]},{"osd":9,"last update":"Fri Jul 14 17:42:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.70699999999999996}]},{"osd":10,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.876}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90700000000000003}]}]},{"osd":5,"up_from":37,"seq":158913789993,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1589248,"data_stored":1059558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47099999999999997,"15min":0.44700000000000001},"min":{"1min":0.35499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77800000000000002,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.48299999999999998,"15min":0.45700000000000002},"min":{"1min":0.35599999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65400000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.52400000000000002}]},{"osd":1,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.55100000000000005,"15min":0.497},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":5.3630000000000004,"5min":5.3630000000000004,"15min":5.3630000000000004},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51300000000000001,"15min":0.48999999999999999},"min":{"1min":0.42399999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.82299999999999995}]},{"osd":2,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.33500000000000002},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.376,"15min":0.35999999999999999},"min":{"1min":0.30499999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.59499999999999997}]},{"osd":3,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.501,"15min":0.495},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.51800000000000002,"15min":0.495},"min":{"1min":0.38700000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.747,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.64500000000000002}]},{"osd":4,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.53500000000000003,"15min":0.50800000000000001},"min":{"1min":0.434,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78100000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54300000000000004,"15min":0.52000000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.72499999999999998}]},{"osd":6,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.78700000000000003,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.54600000000000004,"15min":0.53600000000000003},"min":{"1min":0.45400000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78800000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.68300000000000005}]},{"osd":7,"last update":"Fri Jul 14 17:42:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57099999999999995,"15min":0.56799999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.60599999999999998,"15min":0.60399999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.79900000000000004,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.77500000000000002}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.54700000000000004,"15min":0.54000000000000004},"min":{"1min":0.38100000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.754,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.372,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.876,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.66900000000000004}]},{"osd":9,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.86899999999999999}]},{"osd":10,"last update":"Fri Jul 14 17:42:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":5.3330000000000002,"5min":5.3330000000000002,"15min":5.3330000000000002},"last":0.83999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":40219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":40781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":40593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T17:43:06.018 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-14T17:43:06.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.186+0000 7fa64b084700 1 -- 172.21.15.88:0/481293136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa644100160 msgr2=0x7fa644100570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:06.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.186+0000 7fa64b084700 1 --2- 172.21.15.88:0/481293136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa644100160 0x7fa644100570 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa638009990 tx=0x7fa638009ca0 comp rx=0 tx=0).stop 2023-07-14T17:43:06.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.186+0000 7fa64b084700 1 -- 172.21.15.88:0/481293136 shutdown_connections 2023-07-14T17:43:06.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.186+0000 7fa64b084700 1 --2- 172.21.15.88:0/481293136 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6440ff760 0x7fa644107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:06.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.186+0000 7fa64b084700 1 --2- 172.21.15.88:0/481293136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa644100160 0x7fa644100570 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:06.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.186+0000 7fa64b084700 1 --2- 172.21.15.88:0/481293136 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa6440fedb0 0x7fa6440ff220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:06.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.186+0000 7fa64b084700 1 -- 172.21.15.88:0/481293136 >> 172.21.15.88:0/481293136 conn(0x7fa6440f98d0 msgr2=0x7fa6440fbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:06.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.186+0000 7fa64b084700 1 -- 172.21.15.88:0/481293136 shutdown_connections 2023-07-14T17:43:06.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.186+0000 7fa64b084700 1 -- 172.21.15.88:0/481293136 wait complete. 2023-07-14T17:43:06.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.186+0000 7fa64b084700 1 Processor -- start 2023-07-14T17:43:06.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.186+0000 7fa64b084700 1 -- start start 2023-07-14T17:43:06.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.186+0000 7fa64b084700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6440fedb0 0x7fa644102b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:06.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.186+0000 7fa64b084700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6440ff760 0x7fa6441011a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:06.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.186+0000 7fa64b084700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa644100160 0x7fa6441016e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:06.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.186+0000 7fa64b084700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa644101e80 con 0x7fa6440fedb0 2023-07-14T17:43:06.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.186+0000 7fa64b084700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa644101ff0 con 0x7fa6440ff760 2023-07-14T17:43:06.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.186+0000 7fa64b084700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64419f900 con 0x7fa644100160 2023-07-14T17:43:06.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.186+0000 7fa648e20700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6440fedb0 0x7fa644102b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:06.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.187+0000 7fa648e20700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6440fedb0 0x7fa644102b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42036/0 (socket says 172.21.15.88:42036) 2023-07-14T17:43:06.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.187+0000 7fa648e20700 1 -- 172.21.15.88:0/3310313246 learned_addr learned my addr 172.21.15.88:0/3310313246 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:06.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.187+0000 7fa643fff700 1 --2- 172.21.15.88:0/3310313246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6440ff760 0x7fa6441011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:06.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.187+0000 7fa649621700 1 --2- 172.21.15.88:0/3310313246 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa644100160 0x7fa6441016e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:06.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.187+0000 7fa648e20700 1 -- 172.21.15.88:0/3310313246 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa644100160 msgr2=0x7fa6441016e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:06.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.187+0000 7fa648e20700 1 --2- 172.21.15.88:0/3310313246 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa644100160 0x7fa6441016e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:06.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.187+0000 7fa648e20700 1 -- 172.21.15.88:0/3310313246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6440ff760 msgr2=0x7fa6441011a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:06.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.187+0000 7fa648e20700 1 --2- 172.21.15.88:0/3310313246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6440ff760 0x7fa6441011a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.187+0000 7fa648e20700 1 -- 172.21.15.88:0/3310313246 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6380095f0 con 0x7fa6440fedb0 2023-07-14T17:43:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.187+0000 7fa648e20700 1 --2- 172.21.15.88:0/3310313246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6440fedb0 0x7fa644102b60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa6380059d0 tx=0x7fa638004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.187+0000 7fa641ffb700 1 -- 172.21.15.88:0/3310313246 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63801d070 con 0x7fa6440fedb0 2023-07-14T17:43:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.187+0000 7fa641ffb700 1 -- 172.21.15.88:0/3310313246 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa63800be30 con 0x7fa6440fedb0 2023-07-14T17:43:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.187+0000 7fa641ffb700 1 -- 172.21.15.88:0/3310313246 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa638022db0 con 0x7fa6440fedb0 2023-07-14T17:43:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.187+0000 7fa64b084700 1 -- 172.21.15.88:0/3310313246 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64419fb80 con 0x7fa6440fedb0 2023-07-14T17:43:06.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.187+0000 7fa64b084700 1 -- 172.21.15.88:0/3310313246 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa64419ff50 con 0x7fa6440fedb0 2023-07-14T17:43:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.188+0000 7fa64b084700 1 -- 172.21.15.88:0/3310313246 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64410a1d0 con 0x7fa6440fedb0 2023-07-14T17:43:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.189+0000 7fa641ffb700 1 -- 172.21.15.88:0/3310313246 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa638022570 con 0x7fa6440fedb0 2023-07-14T17:43:06.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.190+0000 7fa641ffb700 1 --2- 172.21.15.88:0/3310313246 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa6340a1220 0x7fa6340a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:06.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.190+0000 7fa643fff700 1 --2- 172.21.15.88:0/3310313246 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa6340a1220 0x7fa6340a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:06.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.190+0000 7fa641ffb700 1 -- 172.21.15.88:0/3310313246 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa6380ce3e0 con 0x7fa6440fedb0 2023-07-14T17:43:06.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.190+0000 7fa643fff700 1 --2- 172.21.15.88:0/3310313246 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa6340a1220 0x7fa6340a36d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa630007be0 tx=0x7fa6300073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:06.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.194+0000 7fa641ffb700 1 -- 172.21.15.88:0/3310313246 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa638097870 con 0x7fa6440fedb0 2023-07-14T17:43:06.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:05 smithi088 ceph-mon[127465]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:43:06.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:05 smithi088 ceph-mon[127465]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:43:06.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:05 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3425644091' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:43:06.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:05 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3004495923' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:43:06.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:05 smithi107 ceph-mon[128999]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:43:06.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:05 smithi107 ceph-mon[128999]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:43:06.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:05 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3425644091' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:43:06.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:05 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3004495923' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:43:06.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.378+0000 7fa64b084700 1 -- 172.21.15.88:0/3310313246 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa6440fe410 con 0x7fa6340a1220 2023-07-14T17:43:06.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.389+0000 7fa641ffb700 1 -- 172.21.15.88:0/3310313246 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+418193 (secure 0 0 0) 0x7fa6440fe410 con 0x7fa6340a1220 2023-07-14T17:43:06.379 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:43:06.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.393+0000 7fa64b084700 1 -- 172.21.15.88:0/3310313246 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa6340a1220 msgr2=0x7fa6340a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:06.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.393+0000 7fa64b084700 1 --2- 172.21.15.88:0/3310313246 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa6340a1220 0x7fa6340a36d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa630007be0 tx=0x7fa6300073d0 comp rx=0 tx=0).stop 2023-07-14T17:43:06.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.393+0000 7fa64b084700 1 -- 172.21.15.88:0/3310313246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6440fedb0 msgr2=0x7fa644102b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:06.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.393+0000 7fa64b084700 1 --2- 172.21.15.88:0/3310313246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6440fedb0 0x7fa644102b60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa6380059d0 tx=0x7fa638004730 comp rx=0 tx=0).stop 2023-07-14T17:43:06.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.393+0000 7fa64b084700 1 -- 172.21.15.88:0/3310313246 shutdown_connections 2023-07-14T17:43:06.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.393+0000 7fa64b084700 1 --2- 172.21.15.88:0/3310313246 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa644100160 0x7fa6441016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:06.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.393+0000 7fa64b084700 1 --2- 172.21.15.88:0/3310313246 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa6340a1220 0x7fa6340a36d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:06.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.393+0000 7fa64b084700 1 --2- 172.21.15.88:0/3310313246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6440fedb0 0x7fa644102b60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:06.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.393+0000 7fa64b084700 1 --2- 172.21.15.88:0/3310313246 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa6440ff760 0x7fa6441011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:06.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.393+0000 7fa64b084700 1 -- 172.21.15.88:0/3310313246 >> 172.21.15.88:0/3310313246 conn(0x7fa6440f98d0 msgr2=0x7fa6440fdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:06.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.393+0000 7fa64b084700 1 -- 172.21.15.88:0/3310313246 shutdown_connections 2023-07-14T17:43:06.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:06.393+0000 7fa64b084700 1 -- 172.21.15.88:0/3310313246 wait complete. 2023-07-14T17:43:06.383 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-07-14T17:43:06.387 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2023-07-14T17:43:05.875883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":634167,"num_objects":84,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":147,"num_read_kb":163,"num_write":369,"num_write_kb":3038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":302,"ondisk_log_size":302,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337424,"kb_used_data":13224,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604400,"statfs":{"total":1151940427776,"available":1151594905600,"internally_reserved":0,"allocated":13541376,"data_stored":7432032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41437,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":21,"num_read_kb":54,"num_write":143,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003020"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.060009+0000","last_change":"2023-07-14T17:42:42.540396+0000","last_active":"2023-07-14T17:42:48.060009+0000","last_peered":"2023-07-14T17:42:48.060009+0000","last_clean":"2023-07-14T17:42:48.060009+0000","last_became_active":"2023-07-14T17:42:42.539732+0000","last_became_peered":"2023-07-14T17:42:42.539732+0000","last_unstale":"2023-07-14T17:42:48.060009+0000","last_undegraded":"2023-07-14T17:42:48.060009+0000","last_fullsized":"2023-07-14T17:42:48.060009+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:15:44.925121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313254+0000","last_change":"2023-07-14T17:42:42.532153+0000","last_active":"2023-07-14T17:42:48.313254+0000","last_peered":"2023-07-14T17:42:48.313254+0000","last_clean":"2023-07-14T17:42:48.313254+0000","last_became_active":"2023-07-14T17:42:42.531745+0000","last_became_peered":"2023-07-14T17:42:42.531745+0000","last_unstale":"2023-07-14T17:42:48.313254+0000","last_undegraded":"2023-07-14T17:42:48.313254+0000","last_fullsized":"2023-07-14T17:42:48.313254+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:55.889251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313184+0000","last_change":"2023-07-14T17:42:42.549497+0000","last_active":"2023-07-14T17:42:48.313184+0000","last_peered":"2023-07-14T17:42:48.313184+0000","last_clean":"2023-07-14T17:42:48.313184+0000","last_became_active":"2023-07-14T17:42:42.549201+0000","last_became_peered":"2023-07-14T17:42:42.549201+0000","last_unstale":"2023-07-14T17:42:48.313184+0000","last_undegraded":"2023-07-14T17:42:48.313184+0000","last_fullsized":"2023-07-14T17:42:48.313184+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:20:16.806348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542180+0000","last_change":"2023-07-14T17:42:42.543460+0000","last_active":"2023-07-14T17:42:47.542180+0000","last_peered":"2023-07-14T17:42:47.542180+0000","last_clean":"2023-07-14T17:42:47.542180+0000","last_became_active":"2023-07-14T17:42:42.543127+0000","last_became_peered":"2023-07-14T17:42:42.543127+0000","last_unstale":"2023-07-14T17:42:47.542180+0000","last_undegraded":"2023-07-14T17:42:47.542180+0000","last_fullsized":"2023-07-14T17:42:47.542180+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:40:04.077779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835392+0000","last_change":"2023-07-14T17:42:42.534242+0000","last_active":"2023-07-14T17:42:47.835392+0000","last_peered":"2023-07-14T17:42:47.835392+0000","last_clean":"2023-07-14T17:42:47.835392+0000","last_became_active":"2023-07-14T17:42:42.533536+0000","last_became_peered":"2023-07-14T17:42:42.533536+0000","last_unstale":"2023-07-14T17:42:47.835392+0000","last_undegraded":"2023-07-14T17:42:47.835392+0000","last_fullsized":"2023-07-14T17:42:47.835392+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:57:39.927450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313767+0000","last_change":"2023-07-14T17:42:42.538042+0000","last_active":"2023-07-14T17:42:48.313767+0000","last_peered":"2023-07-14T17:42:48.313767+0000","last_clean":"2023-07-14T17:42:48.313767+0000","last_became_active":"2023-07-14T17:42:42.537612+0000","last_became_peered":"2023-07-14T17:42:42.537612+0000","last_unstale":"2023-07-14T17:42:48.313767+0000","last_undegraded":"2023-07-14T17:42:48.313767+0000","last_fullsized":"2023-07-14T17:42:48.313767+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:18:57.774815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313672+0000","last_change":"2023-07-14T17:42:42.545318+0000","last_active":"2023-07-14T17:42:48.313672+0000","last_peered":"2023-07-14T17:42:48.313672+0000","last_clean":"2023-07-14T17:42:48.313672+0000","last_became_active":"2023-07-14T17:42:42.544984+0000","last_became_peered":"2023-07-14T17:42:42.544984+0000","last_unstale":"2023-07-14T17:42:48.313672+0000","last_undegraded":"2023-07-14T17:42:48.313672+0000","last_fullsized":"2023-07-14T17:42:48.313672+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:21:01.756500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539255+0000","last_change":"2023-07-14T17:42:42.532609+0000","last_active":"2023-07-14T17:42:47.539255+0000","last_peered":"2023-07-14T17:42:47.539255+0000","last_clean":"2023-07-14T17:42:47.539255+0000","last_became_active":"2023-07-14T17:42:42.532270+0000","last_became_peered":"2023-07-14T17:42:42.532270+0000","last_unstale":"2023-07-14T17:42:47.539255+0000","last_undegraded":"2023-07-14T17:42:47.539255+0000","last_fullsized":"2023-07-14T17:42:47.539255+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:20:00.572052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.315239+0000","last_change":"2023-07-14T17:42:42.548973+0000","last_active":"2023-07-14T17:42:48.315239+0000","last_peered":"2023-07-14T17:42:48.315239+0000","last_clean":"2023-07-14T17:42:48.315239+0000","last_became_active":"2023-07-14T17:42:42.548672+0000","last_became_peered":"2023-07-14T17:42:42.548672+0000","last_unstale":"2023-07-14T17:42:48.315239+0000","last_undegraded":"2023-07-14T17:42:48.315239+0000","last_fullsized":"2023-07-14T17:42:48.315239+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:52:48.887371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.315082+0000","last_change":"2023-07-14T17:42:42.540969+0000","last_active":"2023-07-14T17:42:48.315082+0000","last_peered":"2023-07-14T17:42:48.315082+0000","last_clean":"2023-07-14T17:42:48.315082+0000","last_became_active":"2023-07-14T17:42:42.540653+0000","last_became_peered":"2023-07-14T17:42:42.540653+0000","last_unstale":"2023-07-14T17:42:48.315082+0000","last_undegraded":"2023-07-14T17:42:48.315082+0000","last_fullsized":"2023-07-14T17:42:48.315082+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:18:49.023990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314444+0000","last_change":"2023-07-14T17:42:42.550016+0000","last_active":"2023-07-14T17:42:48.314444+0000","last_peered":"2023-07-14T17:42:48.314444+0000","last_clean":"2023-07-14T17:42:48.314444+0000","last_became_active":"2023-07-14T17:42:42.549653+0000","last_became_peered":"2023-07-14T17:42:42.549653+0000","last_unstale":"2023-07-14T17:42:48.314444+0000","last_undegraded":"2023-07-14T17:42:48.314444+0000","last_fullsized":"2023-07-14T17:42:48.314444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:31:15.876030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542109+0000","last_change":"2023-07-14T17:42:42.552105+0000","last_active":"2023-07-14T17:42:47.542109+0000","last_peered":"2023-07-14T17:42:47.542109+0000","last_clean":"2023-07-14T17:42:47.542109+0000","last_became_active":"2023-07-14T17:42:42.550830+0000","last_became_peered":"2023-07-14T17:42:42.550830+0000","last_unstale":"2023-07-14T17:42:47.542109+0000","last_undegraded":"2023-07-14T17:42:47.542109+0000","last_fullsized":"2023-07-14T17:42:47.542109+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:47:37.896520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313576+0000","last_change":"2023-07-14T17:42:42.534963+0000","last_active":"2023-07-14T17:42:48.313576+0000","last_peered":"2023-07-14T17:42:48.313576+0000","last_clean":"2023-07-14T17:42:48.313576+0000","last_became_active":"2023-07-14T17:42:42.534546+0000","last_became_peered":"2023-07-14T17:42:42.534546+0000","last_unstale":"2023-07-14T17:42:48.313576+0000","last_undegraded":"2023-07-14T17:42:48.313576+0000","last_fullsized":"2023-07-14T17:42:48.313576+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:01:37.492549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.312996+0000","last_change":"2023-07-14T17:42:42.530073+0000","last_active":"2023-07-14T17:42:48.312996+0000","last_peered":"2023-07-14T17:42:48.312996+0000","last_clean":"2023-07-14T17:42:48.312996+0000","last_became_active":"2023-07-14T17:42:42.529767+0000","last_became_peered":"2023-07-14T17:42:42.529767+0000","last_unstale":"2023-07-14T17:42:48.312996+0000","last_undegraded":"2023-07-14T17:42:48.312996+0000","last_fullsized":"2023-07-14T17:42:48.312996+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:13:05.025890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.312868+0000","last_change":"2023-07-14T17:42:42.548168+0000","last_active":"2023-07-14T17:42:48.312868+0000","last_peered":"2023-07-14T17:42:48.312868+0000","last_clean":"2023-07-14T17:42:48.312868+0000","last_became_active":"2023-07-14T17:42:42.547363+0000","last_became_peered":"2023-07-14T17:42:42.547363+0000","last_unstale":"2023-07-14T17:42:48.312868+0000","last_undegraded":"2023-07-14T17:42:48.312868+0000","last_fullsized":"2023-07-14T17:42:48.312868+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:25:15.031613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313218+0000","last_change":"2023-07-14T17:42:42.551042+0000","last_active":"2023-07-14T17:42:48.313218+0000","last_peered":"2023-07-14T17:42:48.313218+0000","last_clean":"2023-07-14T17:42:48.313218+0000","last_became_active":"2023-07-14T17:42:42.550730+0000","last_became_peered":"2023-07-14T17:42:42.550730+0000","last_unstale":"2023-07-14T17:42:48.313218+0000","last_undegraded":"2023-07-14T17:42:48.313218+0000","last_fullsized":"2023-07-14T17:42:48.313218+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:34:57.949192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059890+0000","last_change":"2023-07-14T17:42:42.540528+0000","last_active":"2023-07-14T17:42:48.059890+0000","last_peered":"2023-07-14T17:42:48.059890+0000","last_clean":"2023-07-14T17:42:48.059890+0000","last_became_active":"2023-07-14T17:42:42.540093+0000","last_became_peered":"2023-07-14T17:42:42.540093+0000","last_unstale":"2023-07-14T17:42:48.059890+0000","last_undegraded":"2023-07-14T17:42:48.059890+0000","last_fullsized":"2023-07-14T17:42:48.059890+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:03:51.839812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833464+0000","last_change":"2023-07-14T17:42:42.539512+0000","last_active":"2023-07-14T17:42:47.833464+0000","last_peered":"2023-07-14T17:42:47.833464+0000","last_clean":"2023-07-14T17:42:47.833464+0000","last_became_active":"2023-07-14T17:42:42.539248+0000","last_became_peered":"2023-07-14T17:42:42.539248+0000","last_unstale":"2023-07-14T17:42:47.833464+0000","last_undegraded":"2023-07-14T17:42:47.833464+0000","last_fullsized":"2023-07-14T17:42:47.833464+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:02:47.593908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123669+0000","last_change":"2023-07-14T17:42:42.531630+0000","last_active":"2023-07-14T17:42:48.123669+0000","last_peered":"2023-07-14T17:42:48.123669+0000","last_clean":"2023-07-14T17:42:48.123669+0000","last_became_active":"2023-07-14T17:42:42.531333+0000","last_became_peered":"2023-07-14T17:42:42.531333+0000","last_unstale":"2023-07-14T17:42:48.123669+0000","last_undegraded":"2023-07-14T17:42:48.123669+0000","last_fullsized":"2023-07-14T17:42:48.123669+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:45:40.106913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123071+0000","last_change":"2023-07-14T17:42:46.543062+0000","last_active":"2023-07-14T17:42:48.123071+0000","last_peered":"2023-07-14T17:42:48.123071+0000","last_clean":"2023-07-14T17:42:48.123071+0000","last_became_active":"2023-07-14T17:42:46.542189+0000","last_became_peered":"2023-07-14T17:42:46.542189+0000","last_unstale":"2023-07-14T17:42:48.123071+0000","last_undegraded":"2023-07-14T17:42:48.123071+0000","last_fullsized":"2023-07-14T17:42:48.123071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:42:27.430887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'11","reported_seq":33,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.761444+0000","last_change":"2023-07-14T17:42:41.914589+0000","last_active":"2023-07-14T17:43:02.761444+0000","last_peered":"2023-07-14T17:43:02.761444+0000","last_clean":"2023-07-14T17:43:02.761444+0000","last_became_active":"2023-07-14T17:42:41.914275+0000","last_became_peered":"2023-07-14T17:42:41.914275+0000","last_unstale":"2023-07-14T17:43:02.761444+0000","last_undegraded":"2023-07-14T17:43:02.761444+0000","last_fullsized":"2023-07-14T17:43:02.761444+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:09:10.487178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540020+0000","last_change":"2023-07-14T17:42:42.537207+0000","last_active":"2023-07-14T17:42:47.540020+0000","last_peered":"2023-07-14T17:42:47.540020+0000","last_clean":"2023-07-14T17:42:47.540020+0000","last_became_active":"2023-07-14T17:42:42.536879+0000","last_became_peered":"2023-07-14T17:42:42.536879+0000","last_unstale":"2023-07-14T17:42:47.540020+0000","last_undegraded":"2023-07-14T17:42:47.540020+0000","last_fullsized":"2023-07-14T17:42:47.540020+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:09:29.936504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.541631+0000","last_change":"2023-07-14T17:42:46.542500+0000","last_active":"2023-07-14T17:42:47.541631+0000","last_peered":"2023-07-14T17:42:47.541631+0000","last_clean":"2023-07-14T17:42:47.541631+0000","last_became_active":"2023-07-14T17:42:46.541932+0000","last_became_peered":"2023-07-14T17:42:46.541932+0000","last_unstale":"2023-07-14T17:42:47.541631+0000","last_undegraded":"2023-07-14T17:42:47.541631+0000","last_fullsized":"2023-07-14T17:42:47.541631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:07:53.233373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540068+0000","last_change":"2023-07-14T17:42:41.918169+0000","last_active":"2023-07-14T17:42:47.540068+0000","last_peered":"2023-07-14T17:42:47.540068+0000","last_clean":"2023-07-14T17:42:47.540068+0000","last_became_active":"2023-07-14T17:42:41.917124+0000","last_became_peered":"2023-07-14T17:42:41.917124+0000","last_unstale":"2023-07-14T17:42:47.540068+0000","last_undegraded":"2023-07-14T17:42:47.540068+0000","last_fullsized":"2023-07-14T17:42:47.540068+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:23:35.325653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313826+0000","last_change":"2023-07-14T17:42:42.539265+0000","last_active":"2023-07-14T17:42:48.313826+0000","last_peered":"2023-07-14T17:42:48.313826+0000","last_clean":"2023-07-14T17:42:48.313826+0000","last_became_active":"2023-07-14T17:42:42.538933+0000","last_became_peered":"2023-07-14T17:42:42.538933+0000","last_unstale":"2023-07-14T17:42:48.313826+0000","last_undegraded":"2023-07-14T17:42:48.313826+0000","last_fullsized":"2023-07-14T17:42:48.313826+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:08.277561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542003+0000","last_change":"2023-07-14T17:42:46.550194+0000","last_active":"2023-07-14T17:42:47.542003+0000","last_peered":"2023-07-14T17:42:47.542003+0000","last_clean":"2023-07-14T17:42:47.542003+0000","last_became_active":"2023-07-14T17:42:46.549693+0000","last_became_peered":"2023-07-14T17:42:46.549693+0000","last_unstale":"2023-07-14T17:42:47.542003+0000","last_undegraded":"2023-07-14T17:42:47.542003+0000","last_fullsized":"2023-07-14T17:42:47.542003+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:15.448341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'7","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.762657+0000","last_change":"2023-07-14T17:42:41.918769+0000","last_active":"2023-07-14T17:43:02.762657+0000","last_peered":"2023-07-14T17:43:02.762657+0000","last_clean":"2023-07-14T17:43:02.762657+0000","last_became_active":"2023-07-14T17:42:41.918573+0000","last_became_peered":"2023-07-14T17:42:41.918573+0000","last_unstale":"2023-07-14T17:43:02.762657+0000","last_undegraded":"2023-07-14T17:43:02.762657+0000","last_fullsized":"2023-07-14T17:43:02.762657+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:19:21.409733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.315010+0000","last_change":"2023-07-14T17:42:42.543846+0000","last_active":"2023-07-14T17:42:48.315010+0000","last_peered":"2023-07-14T17:42:48.315010+0000","last_clean":"2023-07-14T17:42:48.315010+0000","last_became_active":"2023-07-14T17:42:42.543528+0000","last_became_peered":"2023-07-14T17:42:42.543528+0000","last_unstale":"2023-07-14T17:42:48.315010+0000","last_undegraded":"2023-07-14T17:42:48.315010+0000","last_fullsized":"2023-07-14T17:42:48.315010+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:12.999288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.541773+0000","last_change":"2023-07-14T17:42:46.547799+0000","last_active":"2023-07-14T17:42:47.541773+0000","last_peered":"2023-07-14T17:42:47.541773+0000","last_clean":"2023-07-14T17:42:47.541773+0000","last_became_active":"2023-07-14T17:42:46.547381+0000","last_became_peered":"2023-07-14T17:42:46.547381+0000","last_unstale":"2023-07-14T17:42:47.541773+0000","last_undegraded":"2023-07-14T17:42:47.541773+0000","last_fullsized":"2023-07-14T17:42:47.541773+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:46:15.220406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835608+0000","last_change":"2023-07-14T17:42:41.914911+0000","last_active":"2023-07-14T17:42:47.835608+0000","last_peered":"2023-07-14T17:42:47.835608+0000","last_clean":"2023-07-14T17:42:47.835608+0000","last_became_active":"2023-07-14T17:42:41.914607+0000","last_became_peered":"2023-07-14T17:42:41.914607+0000","last_unstale":"2023-07-14T17:42:47.835608+0000","last_undegraded":"2023-07-14T17:42:47.835608+0000","last_fullsized":"2023-07-14T17:42:47.835608+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:45.771966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540287+0000","last_change":"2023-07-14T17:42:42.548866+0000","last_active":"2023-07-14T17:42:47.540287+0000","last_peered":"2023-07-14T17:42:47.540287+0000","last_clean":"2023-07-14T17:42:47.540287+0000","last_became_active":"2023-07-14T17:42:42.548562+0000","last_became_peered":"2023-07-14T17:42:42.548562+0000","last_unstale":"2023-07-14T17:42:47.540287+0000","last_undegraded":"2023-07-14T17:42:47.540287+0000","last_fullsized":"2023-07-14T17:42:47.540287+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:56:04.312858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123537+0000","last_change":"2023-07-14T17:42:46.538879+0000","last_active":"2023-07-14T17:42:48.123537+0000","last_peered":"2023-07-14T17:42:48.123537+0000","last_clean":"2023-07-14T17:42:48.123537+0000","last_became_active":"2023-07-14T17:42:46.538476+0000","last_became_peered":"2023-07-14T17:42:46.538476+0000","last_unstale":"2023-07-14T17:42:48.123537+0000","last_undegraded":"2023-07-14T17:42:48.123537+0000","last_fullsized":"2023-07-14T17:42:48.123537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:40:22.866252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.786373+0000","last_change":"2023-07-14T17:42:41.917215+0000","last_active":"2023-07-14T17:43:04.786373+0000","last_peered":"2023-07-14T17:43:04.786373+0000","last_clean":"2023-07-14T17:43:04.786373+0000","last_became_active":"2023-07-14T17:42:41.916640+0000","last_became_peered":"2023-07-14T17:42:41.916640+0000","last_unstale":"2023-07-14T17:43:04.786373+0000","last_undegraded":"2023-07-14T17:43:04.786373+0000","last_fullsized":"2023-07-14T17:43:04.786373+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:37:10.013820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835389+0000","last_change":"2023-07-14T17:42:42.534347+0000","last_active":"2023-07-14T17:42:47.835389+0000","last_peered":"2023-07-14T17:42:47.835389+0000","last_clean":"2023-07-14T17:42:47.835389+0000","last_became_active":"2023-07-14T17:42:42.533958+0000","last_became_peered":"2023-07-14T17:42:42.533958+0000","last_unstale":"2023-07-14T17:42:47.835389+0000","last_undegraded":"2023-07-14T17:42:47.835389+0000","last_fullsized":"2023-07-14T17:42:47.835389+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:52:39.908230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123386+0000","last_change":"2023-07-14T17:42:46.543144+0000","last_active":"2023-07-14T17:42:48.123386+0000","last_peered":"2023-07-14T17:42:48.123386+0000","last_clean":"2023-07-14T17:42:48.123386+0000","last_became_active":"2023-07-14T17:42:46.542311+0000","last_became_peered":"2023-07-14T17:42:46.542311+0000","last_unstale":"2023-07-14T17:42:48.123386+0000","last_undegraded":"2023-07-14T17:42:48.123386+0000","last_fullsized":"2023-07-14T17:42:48.123386+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:05:15.245426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.762300+0000","last_change":"2023-07-14T17:42:41.916174+0000","last_active":"2023-07-14T17:43:02.762300+0000","last_peered":"2023-07-14T17:43:02.762300+0000","last_clean":"2023-07-14T17:43:02.762300+0000","last_became_active":"2023-07-14T17:42:41.915867+0000","last_became_peered":"2023-07-14T17:42:41.915867+0000","last_unstale":"2023-07-14T17:43:02.762300+0000","last_undegraded":"2023-07-14T17:43:02.762300+0000","last_fullsized":"2023-07-14T17:43:02.762300+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:43.000811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540038+0000","last_change":"2023-07-14T17:42:42.547064+0000","last_active":"2023-07-14T17:42:47.540038+0000","last_peered":"2023-07-14T17:42:47.540038+0000","last_clean":"2023-07-14T17:42:47.540038+0000","last_became_active":"2023-07-14T17:42:42.546777+0000","last_became_peered":"2023-07-14T17:42:42.546777+0000","last_unstale":"2023-07-14T17:42:47.540038+0000","last_undegraded":"2023-07-14T17:42:47.540038+0000","last_fullsized":"2023-07-14T17:42:47.540038+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:39:05.720215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059852+0000","last_change":"2023-07-14T17:42:46.548647+0000","last_active":"2023-07-14T17:42:48.059852+0000","last_peered":"2023-07-14T17:42:48.059852+0000","last_clean":"2023-07-14T17:42:48.059852+0000","last_became_active":"2023-07-14T17:42:46.548142+0000","last_became_peered":"2023-07-14T17:42:46.548142+0000","last_unstale":"2023-07-14T17:42:48.059852+0000","last_undegraded":"2023-07-14T17:42:48.059852+0000","last_fullsized":"2023-07-14T17:42:48.059852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:52:44.965835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:01.756003+0000","last_change":"2023-07-14T17:42:41.911675+0000","last_active":"2023-07-14T17:43:01.756003+0000","last_peered":"2023-07-14T17:43:01.756003+0000","last_clean":"2023-07-14T17:43:01.756003+0000","last_became_active":"2023-07-14T17:42:41.911299+0000","last_became_peered":"2023-07-14T17:42:41.911299+0000","last_unstale":"2023-07-14T17:43:01.756003+0000","last_undegraded":"2023-07-14T17:43:01.756003+0000","last_fullsized":"2023-07-14T17:43:01.756003+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:09.880954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539414+0000","last_change":"2023-07-14T17:42:42.536211+0000","last_active":"2023-07-14T17:42:47.539414+0000","last_peered":"2023-07-14T17:42:47.539414+0000","last_clean":"2023-07-14T17:42:47.539414+0000","last_became_active":"2023-07-14T17:42:42.535763+0000","last_became_peered":"2023-07-14T17:42:42.535763+0000","last_unstale":"2023-07-14T17:42:47.539414+0000","last_undegraded":"2023-07-14T17:42:47.539414+0000","last_fullsized":"2023-07-14T17:42:47.539414+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:49:14.800525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313179+0000","last_change":"2023-07-14T17:42:46.540917+0000","last_active":"2023-07-14T17:42:48.313179+0000","last_peered":"2023-07-14T17:42:48.313179+0000","last_clean":"2023-07-14T17:42:48.313179+0000","last_became_active":"2023-07-14T17:42:46.540575+0000","last_became_peered":"2023-07-14T17:42:46.540575+0000","last_unstale":"2023-07-14T17:42:48.313179+0000","last_undegraded":"2023-07-14T17:42:48.313179+0000","last_fullsized":"2023-07-14T17:42:48.313179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:02:20.835521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.771356+0000","last_change":"2023-07-14T17:42:41.917646+0000","last_active":"2023-07-14T17:43:03.771356+0000","last_peered":"2023-07-14T17:43:03.771356+0000","last_clean":"2023-07-14T17:43:03.771356+0000","last_became_active":"2023-07-14T17:42:41.917331+0000","last_became_peered":"2023-07-14T17:42:41.917331+0000","last_unstale":"2023-07-14T17:43:03.771356+0000","last_undegraded":"2023-07-14T17:43:03.771356+0000","last_fullsized":"2023-07-14T17:43:03.771356+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:41:16.425142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833094+0000","last_change":"2023-07-14T17:42:42.526696+0000","last_active":"2023-07-14T17:42:47.833094+0000","last_peered":"2023-07-14T17:42:47.833094+0000","last_clean":"2023-07-14T17:42:47.833094+0000","last_became_active":"2023-07-14T17:42:42.526353+0000","last_became_peered":"2023-07-14T17:42:42.526353+0000","last_unstale":"2023-07-14T17:42:47.833094+0000","last_undegraded":"2023-07-14T17:42:47.833094+0000","last_fullsized":"2023-07-14T17:42:47.833094+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:01:54.943614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539428+0000","last_change":"2023-07-14T17:42:46.545460+0000","last_active":"2023-07-14T17:42:47.539428+0000","last_peered":"2023-07-14T17:42:47.539428+0000","last_clean":"2023-07-14T17:42:47.539428+0000","last_became_active":"2023-07-14T17:42:46.544897+0000","last_became_peered":"2023-07-14T17:42:46.544897+0000","last_unstale":"2023-07-14T17:42:47.539428+0000","last_undegraded":"2023-07-14T17:42:47.539428+0000","last_fullsized":"2023-07-14T17:42:47.539428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:18:28.628297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.782895+0000","last_change":"2023-07-14T17:42:41.912266+0000","last_active":"2023-07-14T17:43:04.782895+0000","last_peered":"2023-07-14T17:43:04.782895+0000","last_clean":"2023-07-14T17:43:04.782895+0000","last_became_active":"2023-07-14T17:42:41.911944+0000","last_became_peered":"2023-07-14T17:42:41.911944+0000","last_unstale":"2023-07-14T17:43:04.782895+0000","last_undegraded":"2023-07-14T17:43:04.782895+0000","last_fullsized":"2023-07-14T17:43:04.782895+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:10:15.152078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.315346+0000","last_change":"2023-07-14T17:42:42.548609+0000","last_active":"2023-07-14T17:42:48.315346+0000","last_peered":"2023-07-14T17:42:48.315346+0000","last_clean":"2023-07-14T17:42:48.315346+0000","last_became_active":"2023-07-14T17:42:42.548253+0000","last_became_peered":"2023-07-14T17:42:42.548253+0000","last_unstale":"2023-07-14T17:42:48.315346+0000","last_undegraded":"2023-07-14T17:42:48.315346+0000","last_fullsized":"2023-07-14T17:42:48.315346+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:02:49.871094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833114+0000","last_change":"2023-07-14T17:42:46.547324+0000","last_active":"2023-07-14T17:42:47.833114+0000","last_peered":"2023-07-14T17:42:47.833114+0000","last_clean":"2023-07-14T17:42:47.833114+0000","last_became_active":"2023-07-14T17:42:46.546876+0000","last_became_peered":"2023-07-14T17:42:46.546876+0000","last_unstale":"2023-07-14T17:42:47.833114+0000","last_undegraded":"2023-07-14T17:42:47.833114+0000","last_fullsized":"2023-07-14T17:42:47.833114+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:30:13.764106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:00.743584+0000","last_change":"2023-07-14T17:42:41.919148+0000","last_active":"2023-07-14T17:43:00.743584+0000","last_peered":"2023-07-14T17:43:00.743584+0000","last_clean":"2023-07-14T17:43:00.743584+0000","last_became_active":"2023-07-14T17:42:41.918842+0000","last_became_peered":"2023-07-14T17:42:41.918842+0000","last_unstale":"2023-07-14T17:43:00.743584+0000","last_undegraded":"2023-07-14T17:43:00.743584+0000","last_fullsized":"2023-07-14T17:43:00.743584+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:04:34.077766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313896+0000","last_change":"2023-07-14T17:42:42.548081+0000","last_active":"2023-07-14T17:42:48.313896+0000","last_peered":"2023-07-14T17:42:48.313896+0000","last_clean":"2023-07-14T17:42:48.313896+0000","last_became_active":"2023-07-14T17:42:42.547876+0000","last_became_peered":"2023-07-14T17:42:42.547876+0000","last_unstale":"2023-07-14T17:42:48.313896+0000","last_undegraded":"2023-07-14T17:42:48.313896+0000","last_fullsized":"2023-07-14T17:42:48.313896+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:53:08.262819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542183+0000","last_change":"2023-07-14T17:42:46.550516+0000","last_active":"2023-07-14T17:42:47.542183+0000","last_peered":"2023-07-14T17:42:47.542183+0000","last_clean":"2023-07-14T17:42:47.542183+0000","last_became_active":"2023-07-14T17:42:46.549588+0000","last_became_peered":"2023-07-14T17:42:46.549588+0000","last_unstale":"2023-07-14T17:42:47.542183+0000","last_undegraded":"2023-07-14T17:42:47.542183+0000","last_fullsized":"2023-07-14T17:42:47.542183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:21.155958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.762568+0000","last_change":"2023-07-14T17:42:41.917527+0000","last_active":"2023-07-14T17:43:02.762568+0000","last_peered":"2023-07-14T17:43:02.762568+0000","last_clean":"2023-07-14T17:43:02.762568+0000","last_became_active":"2023-07-14T17:42:41.917239+0000","last_became_peered":"2023-07-14T17:42:41.917239+0000","last_unstale":"2023-07-14T17:43:02.762568+0000","last_undegraded":"2023-07-14T17:43:02.762568+0000","last_fullsized":"2023-07-14T17:43:02.762568+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:04:18.619214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.315351+0000","last_change":"2023-07-14T17:42:42.554866+0000","last_active":"2023-07-14T17:42:48.315351+0000","last_peered":"2023-07-14T17:42:48.315351+0000","last_clean":"2023-07-14T17:42:48.315351+0000","last_became_active":"2023-07-14T17:42:42.554585+0000","last_became_peered":"2023-07-14T17:42:42.554585+0000","last_unstale":"2023-07-14T17:42:48.315351+0000","last_undegraded":"2023-07-14T17:42:48.315351+0000","last_fullsized":"2023-07-14T17:42:48.315351+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:01.732200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833167+0000","last_change":"2023-07-14T17:42:46.544788+0000","last_active":"2023-07-14T17:42:47.833167+0000","last_peered":"2023-07-14T17:42:47.833167+0000","last_clean":"2023-07-14T17:42:47.833167+0000","last_became_active":"2023-07-14T17:42:46.544419+0000","last_became_peered":"2023-07-14T17:42:46.544419+0000","last_unstale":"2023-07-14T17:42:47.833167+0000","last_undegraded":"2023-07-14T17:42:47.833167+0000","last_fullsized":"2023-07-14T17:42:47.833167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:03:38.940614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"86'24","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.785099+0000","last_change":"2023-07-14T17:42:41.917069+0000","last_active":"2023-07-14T17:43:04.785099+0000","last_peered":"2023-07-14T17:43:04.785099+0000","last_clean":"2023-07-14T17:43:04.785099+0000","last_became_active":"2023-07-14T17:42:41.916630+0000","last_became_peered":"2023-07-14T17:42:41.916630+0000","last_unstale":"2023-07-14T17:43:04.785099+0000","last_undegraded":"2023-07-14T17:43:04.785099+0000","last_fullsized":"2023-07-14T17:43:04.785099+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:05:57.979299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38515,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":39,"num_write":26,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835769+0000","last_change":"2023-07-14T17:42:42.542124+0000","last_active":"2023-07-14T17:42:47.835769+0000","last_peered":"2023-07-14T17:42:47.835769+0000","last_clean":"2023-07-14T17:42:47.835769+0000","last_became_active":"2023-07-14T17:42:42.541851+0000","last_became_peered":"2023-07-14T17:42:42.541851+0000","last_unstale":"2023-07-14T17:42:47.835769+0000","last_undegraded":"2023-07-14T17:42:47.835769+0000","last_fullsized":"2023-07-14T17:42:47.835769+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:06.253283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540680+0000","last_change":"2023-07-14T17:42:46.546167+0000","last_active":"2023-07-14T17:42:47.540680+0000","last_peered":"2023-07-14T17:42:47.540680+0000","last_clean":"2023-07-14T17:42:47.540680+0000","last_became_active":"2023-07-14T17:42:46.545768+0000","last_became_peered":"2023-07-14T17:42:46.545768+0000","last_unstale":"2023-07-14T17:42:47.540680+0000","last_undegraded":"2023-07-14T17:42:47.540680+0000","last_fullsized":"2023-07-14T17:42:47.540680+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:35:53.682315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.781459+0000","last_change":"2023-07-14T17:42:41.918858+0000","last_active":"2023-07-14T17:43:04.781459+0000","last_peered":"2023-07-14T17:43:04.781459+0000","last_clean":"2023-07-14T17:43:04.781459+0000","last_became_active":"2023-07-14T17:42:41.918568+0000","last_became_peered":"2023-07-14T17:42:41.918568+0000","last_unstale":"2023-07-14T17:43:04.781459+0000","last_undegraded":"2023-07-14T17:43:04.781459+0000","last_fullsized":"2023-07-14T17:43:04.781459+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:14:24.080564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313819+0000","last_change":"2023-07-14T17:42:42.542310+0000","last_active":"2023-07-14T17:42:48.313819+0000","last_peered":"2023-07-14T17:42:48.313819+0000","last_clean":"2023-07-14T17:42:48.313819+0000","last_became_active":"2023-07-14T17:42:42.541989+0000","last_became_peered":"2023-07-14T17:42:42.541989+0000","last_unstale":"2023-07-14T17:42:48.313819+0000","last_undegraded":"2023-07-14T17:42:48.313819+0000","last_fullsized":"2023-07-14T17:42:48.313819+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:41:45.629010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059652+0000","last_change":"2023-07-14T17:42:46.548737+0000","last_active":"2023-07-14T17:42:48.059652+0000","last_peered":"2023-07-14T17:42:48.059652+0000","last_clean":"2023-07-14T17:42:48.059652+0000","last_became_active":"2023-07-14T17:42:46.548255+0000","last_became_peered":"2023-07-14T17:42:46.548255+0000","last_unstale":"2023-07-14T17:42:48.059652+0000","last_undegraded":"2023-07-14T17:42:48.059652+0000","last_fullsized":"2023-07-14T17:42:48.059652+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:10.311183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.770445+0000","last_change":"2023-07-14T17:42:41.916382+0000","last_active":"2023-07-14T17:43:03.770445+0000","last_peered":"2023-07-14T17:43:03.770445+0000","last_clean":"2023-07-14T17:43:03.770445+0000","last_became_active":"2023-07-14T17:42:41.916082+0000","last_became_peered":"2023-07-14T17:42:41.916082+0000","last_unstale":"2023-07-14T17:43:03.770445+0000","last_undegraded":"2023-07-14T17:43:03.770445+0000","last_fullsized":"2023-07-14T17:43:03.770445+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:54:15.440758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313921+0000","last_change":"2023-07-14T17:42:42.546812+0000","last_active":"2023-07-14T17:42:48.313921+0000","last_peered":"2023-07-14T17:42:48.313921+0000","last_clean":"2023-07-14T17:42:48.313921+0000","last_became_active":"2023-07-14T17:42:42.546517+0000","last_became_peered":"2023-07-14T17:42:42.546517+0000","last_unstale":"2023-07-14T17:42:48.313921+0000","last_undegraded":"2023-07-14T17:42:48.313921+0000","last_fullsized":"2023-07-14T17:42:48.313921+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:41:22.699813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833687+0000","last_change":"2023-07-14T17:42:46.547726+0000","last_active":"2023-07-14T17:42:47.833687+0000","last_peered":"2023-07-14T17:42:47.833687+0000","last_clean":"2023-07-14T17:42:47.833687+0000","last_became_active":"2023-07-14T17:42:46.547207+0000","last_became_peered":"2023-07-14T17:42:46.547207+0000","last_unstale":"2023-07-14T17:42:47.833687+0000","last_undegraded":"2023-07-14T17:42:47.833687+0000","last_fullsized":"2023-07-14T17:42:47.833687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:55:16.418869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.761166+0000","last_change":"2023-07-14T17:42:41.913802+0000","last_active":"2023-07-14T17:43:02.761166+0000","last_peered":"2023-07-14T17:43:02.761166+0000","last_clean":"2023-07-14T17:43:02.761166+0000","last_became_active":"2023-07-14T17:42:41.913467+0000","last_became_peered":"2023-07-14T17:42:41.913467+0000","last_unstale":"2023-07-14T17:43:02.761166+0000","last_undegraded":"2023-07-14T17:43:02.761166+0000","last_fullsized":"2023-07-14T17:43:02.761166+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:46:59.725226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313504+0000","last_change":"2023-07-14T17:42:42.545379+0000","last_active":"2023-07-14T17:42:48.313504+0000","last_peered":"2023-07-14T17:42:48.313504+0000","last_clean":"2023-07-14T17:42:48.313504+0000","last_became_active":"2023-07-14T17:42:42.545012+0000","last_became_peered":"2023-07-14T17:42:42.545012+0000","last_unstale":"2023-07-14T17:42:48.313504+0000","last_undegraded":"2023-07-14T17:42:48.313504+0000","last_fullsized":"2023-07-14T17:42:48.313504+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:10.885034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313517+0000","last_change":"2023-07-14T17:42:46.544524+0000","last_active":"2023-07-14T17:42:48.313517+0000","last_peered":"2023-07-14T17:42:48.313517+0000","last_clean":"2023-07-14T17:42:48.313517+0000","last_became_active":"2023-07-14T17:42:46.544152+0000","last_became_peered":"2023-07-14T17:42:46.544152+0000","last_unstale":"2023-07-14T17:42:48.313517+0000","last_undegraded":"2023-07-14T17:42:48.313517+0000","last_fullsized":"2023-07-14T17:42:48.313517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:01:14.398239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'16","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:01.756350+0000","last_change":"2023-07-14T17:42:41.917282+0000","last_active":"2023-07-14T17:43:01.756350+0000","last_peered":"2023-07-14T17:43:01.756350+0000","last_clean":"2023-07-14T17:43:01.756350+0000","last_became_active":"2023-07-14T17:42:41.916994+0000","last_became_peered":"2023-07-14T17:42:41.916994+0000","last_unstale":"2023-07-14T17:43:01.756350+0000","last_undegraded":"2023-07-14T17:43:01.756350+0000","last_fullsized":"2023-07-14T17:43:01.756350+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:41:27.420237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540359+0000","last_change":"2023-07-14T17:42:42.548525+0000","last_active":"2023-07-14T17:42:47.540359+0000","last_peered":"2023-07-14T17:42:47.540359+0000","last_clean":"2023-07-14T17:42:47.540359+0000","last_became_active":"2023-07-14T17:42:42.548270+0000","last_became_peered":"2023-07-14T17:42:42.548270+0000","last_unstale":"2023-07-14T17:42:47.540359+0000","last_undegraded":"2023-07-14T17:42:47.540359+0000","last_fullsized":"2023-07-14T17:42:47.540359+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:18:32.383297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540365+0000","last_change":"2023-07-14T17:42:46.546480+0000","last_active":"2023-07-14T17:42:47.540365+0000","last_peered":"2023-07-14T17:42:47.540365+0000","last_clean":"2023-07-14T17:42:47.540365+0000","last_became_active":"2023-07-14T17:42:46.546078+0000","last_became_peered":"2023-07-14T17:42:46.546078+0000","last_unstale":"2023-07-14T17:42:47.540365+0000","last_undegraded":"2023-07-14T17:42:47.540365+0000","last_fullsized":"2023-07-14T17:42:47.540365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:17:38.547810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835965+0000","last_change":"2023-07-14T17:42:41.914226+0000","last_active":"2023-07-14T17:42:47.835965+0000","last_peered":"2023-07-14T17:42:47.835965+0000","last_clean":"2023-07-14T17:42:47.835965+0000","last_became_active":"2023-07-14T17:42:41.913820+0000","last_became_peered":"2023-07-14T17:42:41.913820+0000","last_unstale":"2023-07-14T17:42:47.835965+0000","last_undegraded":"2023-07-14T17:42:47.835965+0000","last_fullsized":"2023-07-14T17:42:47.835965+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:31:36.040725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059864+0000","last_change":"2023-07-14T17:42:42.543549+0000","last_active":"2023-07-14T17:42:48.059864+0000","last_peered":"2023-07-14T17:42:48.059864+0000","last_clean":"2023-07-14T17:42:48.059864+0000","last_became_active":"2023-07-14T17:42:42.543245+0000","last_became_peered":"2023-07-14T17:42:42.543245+0000","last_unstale":"2023-07-14T17:42:48.059864+0000","last_undegraded":"2023-07-14T17:42:48.059864+0000","last_fullsized":"2023-07-14T17:42:48.059864+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:30:07.889057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835441+0000","last_change":"2023-07-14T17:42:46.577245+0000","last_active":"2023-07-14T17:42:47.835441+0000","last_peered":"2023-07-14T17:42:47.835441+0000","last_clean":"2023-07-14T17:42:47.835441+0000","last_became_active":"2023-07-14T17:42:46.576793+0000","last_became_peered":"2023-07-14T17:42:46.576793+0000","last_unstale":"2023-07-14T17:42:47.835441+0000","last_undegraded":"2023-07-14T17:42:47.835441+0000","last_fullsized":"2023-07-14T17:42:47.835441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:29:32.688546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833389+0000","last_change":"2023-07-14T17:42:41.917199+0000","last_active":"2023-07-14T17:42:47.833389+0000","last_peered":"2023-07-14T17:42:47.833389+0000","last_clean":"2023-07-14T17:42:47.833389+0000","last_became_active":"2023-07-14T17:42:41.916924+0000","last_became_peered":"2023-07-14T17:42:41.916924+0000","last_unstale":"2023-07-14T17:42:47.833389+0000","last_undegraded":"2023-07-14T17:42:47.833389+0000","last_fullsized":"2023-07-14T17:42:47.833389+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:12:02.515835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314451+0000","last_change":"2023-07-14T17:42:42.540531+0000","last_active":"2023-07-14T17:42:48.314451+0000","last_peered":"2023-07-14T17:42:48.314451+0000","last_clean":"2023-07-14T17:42:48.314451+0000","last_became_active":"2023-07-14T17:42:42.540165+0000","last_became_peered":"2023-07-14T17:42:42.540165+0000","last_unstale":"2023-07-14T17:42:48.314451+0000","last_undegraded":"2023-07-14T17:42:48.314451+0000","last_fullsized":"2023-07-14T17:42:48.314451+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:34:21.411165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313334+0000","last_change":"2023-07-14T17:42:46.537158+0000","last_active":"2023-07-14T17:42:48.313334+0000","last_peered":"2023-07-14T17:42:48.313334+0000","last_clean":"2023-07-14T17:42:48.313334+0000","last_became_active":"2023-07-14T17:42:46.536816+0000","last_became_peered":"2023-07-14T17:42:46.536816+0000","last_unstale":"2023-07-14T17:42:48.313334+0000","last_undegraded":"2023-07-14T17:42:48.313334+0000","last_fullsized":"2023-07-14T17:42:48.313334+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:47:53.565328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313911+0000","last_change":"2023-07-14T17:42:41.913638+0000","last_active":"2023-07-14T17:42:48.313911+0000","last_peered":"2023-07-14T17:42:48.313911+0000","last_clean":"2023-07-14T17:42:48.313911+0000","last_became_active":"2023-07-14T17:42:41.913361+0000","last_became_peered":"2023-07-14T17:42:41.913361+0000","last_unstale":"2023-07-14T17:42:48.313911+0000","last_undegraded":"2023-07-14T17:42:48.313911+0000","last_fullsized":"2023-07-14T17:42:48.313911+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:30:25.777267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314326+0000","last_change":"2023-07-14T17:42:42.541738+0000","last_active":"2023-07-14T17:42:48.314326+0000","last_peered":"2023-07-14T17:42:48.314326+0000","last_clean":"2023-07-14T17:42:48.314326+0000","last_became_active":"2023-07-14T17:42:42.541264+0000","last_became_peered":"2023-07-14T17:42:42.541264+0000","last_unstale":"2023-07-14T17:42:48.314326+0000","last_undegraded":"2023-07-14T17:42:48.314326+0000","last_fullsized":"2023-07-14T17:42:48.314326+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:38:28.774135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313008+0000","last_change":"2023-07-14T17:42:46.576999+0000","last_active":"2023-07-14T17:42:48.313008+0000","last_peered":"2023-07-14T17:42:48.313008+0000","last_clean":"2023-07-14T17:42:48.313008+0000","last_became_active":"2023-07-14T17:42:46.576652+0000","last_became_peered":"2023-07-14T17:42:46.576652+0000","last_unstale":"2023-07-14T17:42:48.313008+0000","last_undegraded":"2023-07-14T17:42:48.313008+0000","last_fullsized":"2023-07-14T17:42:48.313008+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:59:23.913753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540039+0000","last_change":"2023-07-14T17:42:42.547053+0000","last_active":"2023-07-14T17:42:47.540039+0000","last_peered":"2023-07-14T17:42:47.540039+0000","last_clean":"2023-07-14T17:42:47.540039+0000","last_became_active":"2023-07-14T17:42:42.546735+0000","last_became_peered":"2023-07-14T17:42:42.546735+0000","last_unstale":"2023-07-14T17:42:47.540039+0000","last_undegraded":"2023-07-14T17:42:47.540039+0000","last_fullsized":"2023-07-14T17:42:47.540039+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:19:26.849479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.781254+0000","last_change":"2023-07-14T17:42:41.918692+0000","last_active":"2023-07-14T17:43:04.781254+0000","last_peered":"2023-07-14T17:43:04.781254+0000","last_clean":"2023-07-14T17:43:04.781254+0000","last_became_active":"2023-07-14T17:42:41.918360+0000","last_became_peered":"2023-07-14T17:42:41.918360+0000","last_unstale":"2023-07-14T17:43:04.781254+0000","last_undegraded":"2023-07-14T17:43:04.781254+0000","last_fullsized":"2023-07-14T17:43:04.781254+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:40:45.927843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314217+0000","last_change":"2023-07-14T17:42:46.543997+0000","last_active":"2023-07-14T17:42:48.314217+0000","last_peered":"2023-07-14T17:42:48.314217+0000","last_clean":"2023-07-14T17:42:48.314217+0000","last_became_active":"2023-07-14T17:42:46.543650+0000","last_became_peered":"2023-07-14T17:42:46.543650+0000","last_unstale":"2023-07-14T17:42:48.314217+0000","last_undegraded":"2023-07-14T17:42:48.314217+0000","last_fullsized":"2023-07-14T17:42:48.314217+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:25:35.518988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539400+0000","last_change":"2023-07-14T17:42:42.548860+0000","last_active":"2023-07-14T17:42:47.539400+0000","last_peered":"2023-07-14T17:42:47.539400+0000","last_clean":"2023-07-14T17:42:47.539400+0000","last_became_active":"2023-07-14T17:42:42.548564+0000","last_became_peered":"2023-07-14T17:42:42.548564+0000","last_unstale":"2023-07-14T17:42:47.539400+0000","last_undegraded":"2023-07-14T17:42:47.539400+0000","last_fullsized":"2023-07-14T17:42:47.539400+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:07:45.387117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'5","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.761783+0000","last_change":"2023-07-14T17:42:41.912601+0000","last_active":"2023-07-14T17:43:02.761783+0000","last_peered":"2023-07-14T17:43:02.761783+0000","last_clean":"2023-07-14T17:43:02.761783+0000","last_became_active":"2023-07-14T17:42:41.912265+0000","last_became_peered":"2023-07-14T17:42:41.912265+0000","last_unstale":"2023-07-14T17:43:02.761783+0000","last_undegraded":"2023-07-14T17:43:02.761783+0000","last_fullsized":"2023-07-14T17:43:02.761783+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:58.872606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":343,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.312950+0000","last_change":"2023-07-14T17:41:43.696093+0000","last_active":"2023-07-14T17:42:48.312950+0000","last_peered":"2023-07-14T17:42:48.312950+0000","last_clean":"2023-07-14T17:42:48.312950+0000","last_became_active":"2023-07-14T17:41:43.695511+0000","last_became_peered":"2023-07-14T17:41:43.695511+0000","last_unstale":"2023-07-14T17:42:48.312950+0000","last_undegraded":"2023-07-14T17:42:48.312950+0000","last_fullsized":"2023-07-14T17:42:48.312950+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:38:50.657865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:38:50.657865+0000","last_clean_scrub_stamp":"2023-07-14T17:38:50.657865+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:30:55.242715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540596+0000","last_change":"2023-07-14T17:42:46.577540+0000","last_active":"2023-07-14T17:42:47.540596+0000","last_peered":"2023-07-14T17:42:47.540596+0000","last_clean":"2023-07-14T17:42:47.540596+0000","last_became_active":"2023-07-14T17:42:46.577067+0000","last_became_peered":"2023-07-14T17:42:46.577067+0000","last_unstale":"2023-07-14T17:42:47.540596+0000","last_undegraded":"2023-07-14T17:42:47.540596+0000","last_fullsized":"2023-07-14T17:42:47.540596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:39:25.538930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313453+0000","last_change":"2023-07-14T17:42:42.543243+0000","last_active":"2023-07-14T17:42:48.313453+0000","last_peered":"2023-07-14T17:42:48.313453+0000","last_clean":"2023-07-14T17:42:48.313453+0000","last_became_active":"2023-07-14T17:42:42.542965+0000","last_became_peered":"2023-07-14T17:42:42.542965+0000","last_unstale":"2023-07-14T17:42:48.313453+0000","last_undegraded":"2023-07-14T17:42:48.313453+0000","last_fullsized":"2023-07-14T17:42:48.313453+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:08:21.008138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313410+0000","last_change":"2023-07-14T17:42:41.920806+0000","last_active":"2023-07-14T17:42:48.313410+0000","last_peered":"2023-07-14T17:42:48.313410+0000","last_clean":"2023-07-14T17:42:48.313410+0000","last_became_active":"2023-07-14T17:42:41.920528+0000","last_became_peered":"2023-07-14T17:42:41.920528+0000","last_unstale":"2023-07-14T17:42:48.313410+0000","last_undegraded":"2023-07-14T17:42:48.313410+0000","last_fullsized":"2023-07-14T17:42:48.313410+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:43:43.408841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059657+0000","last_change":"2023-07-14T17:42:46.542026+0000","last_active":"2023-07-14T17:42:48.059657+0000","last_peered":"2023-07-14T17:42:48.059657+0000","last_clean":"2023-07-14T17:42:48.059657+0000","last_became_active":"2023-07-14T17:42:46.541672+0000","last_became_peered":"2023-07-14T17:42:46.541672+0000","last_unstale":"2023-07-14T17:42:48.059657+0000","last_undegraded":"2023-07-14T17:42:48.059657+0000","last_fullsized":"2023-07-14T17:42:48.059657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:45:35.131993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"86'3","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.761841+0000","last_change":"2023-07-14T17:42:41.915798+0000","last_active":"2023-07-14T17:43:02.761841+0000","last_peered":"2023-07-14T17:43:02.761841+0000","last_clean":"2023-07-14T17:43:02.761841+0000","last_became_active":"2023-07-14T17:42:41.915492+0000","last_became_peered":"2023-07-14T17:42:41.915492+0000","last_unstale":"2023-07-14T17:43:02.761841+0000","last_undegraded":"2023-07-14T17:43:02.761841+0000","last_fullsized":"2023-07-14T17:43:02.761841+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:25:42.031839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313564+0000","last_change":"2023-07-14T17:42:42.533500+0000","last_active":"2023-07-14T17:42:48.313564+0000","last_peered":"2023-07-14T17:42:48.313564+0000","last_clean":"2023-07-14T17:42:48.313564+0000","last_became_active":"2023-07-14T17:42:42.533232+0000","last_became_peered":"2023-07-14T17:42:42.533232+0000","last_unstale":"2023-07-14T17:42:48.313564+0000","last_undegraded":"2023-07-14T17:42:48.313564+0000","last_fullsized":"2023-07-14T17:42:48.313564+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:57:14.212427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539366+0000","last_change":"2023-07-14T17:42:46.577131+0000","last_active":"2023-07-14T17:42:47.539366+0000","last_peered":"2023-07-14T17:42:47.539366+0000","last_clean":"2023-07-14T17:42:47.539366+0000","last_became_active":"2023-07-14T17:42:46.576695+0000","last_became_peered":"2023-07-14T17:42:46.576695+0000","last_unstale":"2023-07-14T17:42:47.539366+0000","last_undegraded":"2023-07-14T17:42:47.539366+0000","last_fullsized":"2023-07-14T17:42:47.539366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:49:05.554462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'7","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.762829+0000","last_change":"2023-07-14T17:42:41.919021+0000","last_active":"2023-07-14T17:43:02.762829+0000","last_peered":"2023-07-14T17:43:02.762829+0000","last_clean":"2023-07-14T17:43:02.762829+0000","last_became_active":"2023-07-14T17:42:41.918714+0000","last_became_peered":"2023-07-14T17:42:41.918714+0000","last_unstale":"2023-07-14T17:43:02.762829+0000","last_undegraded":"2023-07-14T17:43:02.762829+0000","last_fullsized":"2023-07-14T17:43:02.762829+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:40:48.400679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539836+0000","last_change":"2023-07-14T17:42:42.534661+0000","last_active":"2023-07-14T17:42:47.539836+0000","last_peered":"2023-07-14T17:42:47.539836+0000","last_clean":"2023-07-14T17:42:47.539836+0000","last_became_active":"2023-07-14T17:42:42.534357+0000","last_became_peered":"2023-07-14T17:42:42.534357+0000","last_unstale":"2023-07-14T17:42:47.539836+0000","last_undegraded":"2023-07-14T17:42:47.539836+0000","last_fullsized":"2023-07-14T17:42:47.539836+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:24:53.645011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.122946+0000","last_change":"2023-07-14T17:42:46.538648+0000","last_active":"2023-07-14T17:42:48.122946+0000","last_peered":"2023-07-14T17:42:48.122946+0000","last_clean":"2023-07-14T17:42:48.122946+0000","last_became_active":"2023-07-14T17:42:46.538225+0000","last_became_peered":"2023-07-14T17:42:46.538225+0000","last_unstale":"2023-07-14T17:42:48.122946+0000","last_undegraded":"2023-07-14T17:42:48.122946+0000","last_fullsized":"2023-07-14T17:42:48.122946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:36:57.320814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'5","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:00.743421+0000","last_change":"2023-07-14T17:42:41.915406+0000","last_active":"2023-07-14T17:43:00.743421+0000","last_peered":"2023-07-14T17:43:00.743421+0000","last_clean":"2023-07-14T17:43:00.743421+0000","last_became_active":"2023-07-14T17:42:41.914967+0000","last_became_peered":"2023-07-14T17:42:41.914967+0000","last_unstale":"2023-07-14T17:43:00.743421+0000","last_undegraded":"2023-07-14T17:43:00.743421+0000","last_fullsized":"2023-07-14T17:43:00.743421+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:56:07.113627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313341+0000","last_change":"2023-07-14T17:42:42.539052+0000","last_active":"2023-07-14T17:42:48.313341+0000","last_peered":"2023-07-14T17:42:48.313341+0000","last_clean":"2023-07-14T17:42:48.313341+0000","last_became_active":"2023-07-14T17:42:42.538752+0000","last_became_peered":"2023-07-14T17:42:42.538752+0000","last_unstale":"2023-07-14T17:42:48.313341+0000","last_undegraded":"2023-07-14T17:42:48.313341+0000","last_fullsized":"2023-07-14T17:42:48.313341+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:25:31.544906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542387+0000","last_change":"2023-07-14T17:42:46.577224+0000","last_active":"2023-07-14T17:42:47.542387+0000","last_peered":"2023-07-14T17:42:47.542387+0000","last_clean":"2023-07-14T17:42:47.542387+0000","last_became_active":"2023-07-14T17:42:46.576781+0000","last_became_peered":"2023-07-14T17:42:46.576781+0000","last_unstale":"2023-07-14T17:42:47.542387+0000","last_undegraded":"2023-07-14T17:42:47.542387+0000","last_fullsized":"2023-07-14T17:42:47.542387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:47:23.639743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:00.744812+0000","last_change":"2023-07-14T17:42:41.916406+0000","last_active":"2023-07-14T17:43:00.744812+0000","last_peered":"2023-07-14T17:43:00.744812+0000","last_clean":"2023-07-14T17:43:00.744812+0000","last_became_active":"2023-07-14T17:42:41.916115+0000","last_became_peered":"2023-07-14T17:42:41.916115+0000","last_unstale":"2023-07-14T17:43:00.744812+0000","last_undegraded":"2023-07-14T17:43:00.744812+0000","last_fullsized":"2023-07-14T17:43:00.744812+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:59:35.868770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835280+0000","last_change":"2023-07-14T17:42:42.529969+0000","last_active":"2023-07-14T17:42:47.835280+0000","last_peered":"2023-07-14T17:42:47.835280+0000","last_clean":"2023-07-14T17:42:47.835280+0000","last_became_active":"2023-07-14T17:42:42.529659+0000","last_became_peered":"2023-07-14T17:42:42.529659+0000","last_unstale":"2023-07-14T17:42:47.835280+0000","last_undegraded":"2023-07-14T17:42:47.835280+0000","last_fullsized":"2023-07-14T17:42:47.835280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:45:58.093667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833742+0000","last_change":"2023-07-14T17:42:46.546303+0000","last_active":"2023-07-14T17:42:47.833742+0000","last_peered":"2023-07-14T17:42:47.833742+0000","last_clean":"2023-07-14T17:42:47.833742+0000","last_became_active":"2023-07-14T17:42:46.545947+0000","last_became_peered":"2023-07-14T17:42:46.545947+0000","last_unstale":"2023-07-14T17:42:47.833742+0000","last_undegraded":"2023-07-14T17:42:47.833742+0000","last_fullsized":"2023-07-14T17:42:47.833742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:08:49.113221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"86'5","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.761458+0000","last_change":"2023-07-14T17:42:41.916373+0000","last_active":"2023-07-14T17:43:02.761458+0000","last_peered":"2023-07-14T17:43:02.761458+0000","last_clean":"2023-07-14T17:43:02.761458+0000","last_became_active":"2023-07-14T17:42:41.915966+0000","last_became_peered":"2023-07-14T17:42:41.915966+0000","last_unstale":"2023-07-14T17:43:02.761458+0000","last_undegraded":"2023-07-14T17:43:02.761458+0000","last_fullsized":"2023-07-14T17:43:02.761458+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:40:26.456139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059423+0000","last_change":"2023-07-14T17:42:42.532040+0000","last_active":"2023-07-14T17:42:48.059423+0000","last_peered":"2023-07-14T17:42:48.059423+0000","last_clean":"2023-07-14T17:42:48.059423+0000","last_became_active":"2023-07-14T17:42:42.531690+0000","last_became_peered":"2023-07-14T17:42:42.531690+0000","last_unstale":"2023-07-14T17:42:48.059423+0000","last_undegraded":"2023-07-14T17:42:48.059423+0000","last_fullsized":"2023-07-14T17:42:48.059423+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:36:37.658897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542001+0000","last_change":"2023-07-14T17:42:46.550166+0000","last_active":"2023-07-14T17:42:47.542001+0000","last_peered":"2023-07-14T17:42:47.542001+0000","last_clean":"2023-07-14T17:42:47.542001+0000","last_became_active":"2023-07-14T17:42:46.549700+0000","last_became_peered":"2023-07-14T17:42:46.549700+0000","last_unstale":"2023-07-14T17:42:47.542001+0000","last_undegraded":"2023-07-14T17:42:47.542001+0000","last_fullsized":"2023-07-14T17:42:47.542001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:23.561271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"86'2","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.762988+0000","last_change":"2023-07-14T17:42:41.917309+0000","last_active":"2023-07-14T17:43:02.762988+0000","last_peered":"2023-07-14T17:43:02.762988+0000","last_clean":"2023-07-14T17:43:02.762988+0000","last_became_active":"2023-07-14T17:42:41.916694+0000","last_became_peered":"2023-07-14T17:42:41.916694+0000","last_unstale":"2023-07-14T17:43:02.762988+0000","last_undegraded":"2023-07-14T17:43:02.762988+0000","last_fullsized":"2023-07-14T17:43:02.762988+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:12:41.209743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539969+0000","last_change":"2023-07-14T17:42:42.531789+0000","last_active":"2023-07-14T17:42:47.539969+0000","last_peered":"2023-07-14T17:42:47.539969+0000","last_clean":"2023-07-14T17:42:47.539969+0000","last_became_active":"2023-07-14T17:42:42.531460+0000","last_became_peered":"2023-07-14T17:42:42.531460+0000","last_unstale":"2023-07-14T17:42:47.539969+0000","last_undegraded":"2023-07-14T17:42:47.539969+0000","last_fullsized":"2023-07-14T17:42:47.539969+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:17.645748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.541998+0000","last_change":"2023-07-14T17:42:46.577034+0000","last_active":"2023-07-14T17:42:47.541998+0000","last_peered":"2023-07-14T17:42:47.541998+0000","last_clean":"2023-07-14T17:42:47.541998+0000","last_became_active":"2023-07-14T17:42:46.576636+0000","last_became_peered":"2023-07-14T17:42:46.576636+0000","last_unstale":"2023-07-14T17:42:47.541998+0000","last_undegraded":"2023-07-14T17:42:47.541998+0000","last_fullsized":"2023-07-14T17:42:47.541998+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:08:40.397509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:00.743144+0000","last_change":"2023-07-14T17:42:41.919099+0000","last_active":"2023-07-14T17:43:00.743144+0000","last_peered":"2023-07-14T17:43:00.743144+0000","last_clean":"2023-07-14T17:43:00.743144+0000","last_became_active":"2023-07-14T17:42:41.918306+0000","last_became_peered":"2023-07-14T17:42:41.918306+0000","last_unstale":"2023-07-14T17:43:00.743144+0000","last_undegraded":"2023-07-14T17:43:00.743144+0000","last_fullsized":"2023-07-14T17:43:00.743144+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:20:09.398489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314097+0000","last_change":"2023-07-14T17:42:42.552969+0000","last_active":"2023-07-14T17:42:48.314097+0000","last_peered":"2023-07-14T17:42:48.314097+0000","last_clean":"2023-07-14T17:42:48.314097+0000","last_became_active":"2023-07-14T17:42:42.552647+0000","last_became_peered":"2023-07-14T17:42:42.552647+0000","last_unstale":"2023-07-14T17:42:48.314097+0000","last_undegraded":"2023-07-14T17:42:48.314097+0000","last_fullsized":"2023-07-14T17:42:48.314097+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:24:37.409124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314884+0000","last_change":"2023-07-14T17:42:46.538335+0000","last_active":"2023-07-14T17:42:48.314884+0000","last_peered":"2023-07-14T17:42:48.314884+0000","last_clean":"2023-07-14T17:42:48.314884+0000","last_became_active":"2023-07-14T17:42:46.537929+0000","last_became_peered":"2023-07-14T17:42:46.537929+0000","last_unstale":"2023-07-14T17:42:48.314884+0000","last_undegraded":"2023-07-14T17:42:48.314884+0000","last_fullsized":"2023-07-14T17:42:48.314884+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:42:43.435298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.785078+0000","last_change":"2023-07-14T17:42:41.916323+0000","last_active":"2023-07-14T17:43:04.785078+0000","last_peered":"2023-07-14T17:43:04.785078+0000","last_clean":"2023-07-14T17:43:04.785078+0000","last_became_active":"2023-07-14T17:42:41.916019+0000","last_became_peered":"2023-07-14T17:42:41.916019+0000","last_unstale":"2023-07-14T17:43:04.785078+0000","last_undegraded":"2023-07-14T17:43:04.785078+0000","last_fullsized":"2023-07-14T17:43:04.785078+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:42:06.067793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314443+0000","last_change":"2023-07-14T17:42:42.549843+0000","last_active":"2023-07-14T17:42:48.314443+0000","last_peered":"2023-07-14T17:42:48.314443+0000","last_clean":"2023-07-14T17:42:48.314443+0000","last_became_active":"2023-07-14T17:42:42.549347+0000","last_became_peered":"2023-07-14T17:42:42.549347+0000","last_unstale":"2023-07-14T17:42:48.314443+0000","last_undegraded":"2023-07-14T17:42:48.314443+0000","last_fullsized":"2023-07-14T17:42:48.314443+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:37:33.281681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313679+0000","last_change":"2023-07-14T17:42:46.545262+0000","last_active":"2023-07-14T17:42:48.313679+0000","last_peered":"2023-07-14T17:42:48.313679+0000","last_clean":"2023-07-14T17:42:48.313679+0000","last_became_active":"2023-07-14T17:42:46.544891+0000","last_became_peered":"2023-07-14T17:42:46.544891+0000","last_unstale":"2023-07-14T17:42:48.313679+0000","last_undegraded":"2023-07-14T17:42:48.313679+0000","last_fullsized":"2023-07-14T17:42:48.313679+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:58:10.135808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.770619+0000","last_change":"2023-07-14T17:42:41.912855+0000","last_active":"2023-07-14T17:43:03.770619+0000","last_peered":"2023-07-14T17:43:03.770619+0000","last_clean":"2023-07-14T17:43:03.770619+0000","last_became_active":"2023-07-14T17:42:41.912510+0000","last_became_peered":"2023-07-14T17:42:41.912510+0000","last_unstale":"2023-07-14T17:43:03.770619+0000","last_undegraded":"2023-07-14T17:43:03.770619+0000","last_fullsized":"2023-07-14T17:43:03.770619+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:58:29.251068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313890+0000","last_change":"2023-07-14T17:42:42.545344+0000","last_active":"2023-07-14T17:42:48.313890+0000","last_peered":"2023-07-14T17:42:48.313890+0000","last_clean":"2023-07-14T17:42:48.313890+0000","last_became_active":"2023-07-14T17:42:42.545056+0000","last_became_peered":"2023-07-14T17:42:42.545056+0000","last_unstale":"2023-07-14T17:42:48.313890+0000","last_undegraded":"2023-07-14T17:42:48.313890+0000","last_fullsized":"2023-07-14T17:42:48.313890+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:36:25.864767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539658+0000","last_change":"2023-07-14T17:42:46.541279+0000","last_active":"2023-07-14T17:42:47.539658+0000","last_peered":"2023-07-14T17:42:47.539658+0000","last_clean":"2023-07-14T17:42:47.539658+0000","last_became_active":"2023-07-14T17:42:46.540835+0000","last_became_peered":"2023-07-14T17:42:46.540835+0000","last_unstale":"2023-07-14T17:42:47.539658+0000","last_undegraded":"2023-07-14T17:42:47.539658+0000","last_fullsized":"2023-07-14T17:42:47.539658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:04:21.200711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.771177+0000","last_change":"2023-07-14T17:42:41.919257+0000","last_active":"2023-07-14T17:43:03.771177+0000","last_peered":"2023-07-14T17:43:03.771177+0000","last_clean":"2023-07-14T17:43:03.771177+0000","last_became_active":"2023-07-14T17:42:41.918968+0000","last_became_peered":"2023-07-14T17:42:41.918968+0000","last_unstale":"2023-07-14T17:43:03.771177+0000","last_undegraded":"2023-07-14T17:43:03.771177+0000","last_fullsized":"2023-07-14T17:43:03.771177+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:16:49.994602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313885+0000","last_change":"2023-07-14T17:42:42.543993+0000","last_active":"2023-07-14T17:42:48.313885+0000","last_peered":"2023-07-14T17:42:48.313885+0000","last_clean":"2023-07-14T17:42:48.313885+0000","last_became_active":"2023-07-14T17:42:42.543685+0000","last_became_peered":"2023-07-14T17:42:42.543685+0000","last_unstale":"2023-07-14T17:42:48.313885+0000","last_undegraded":"2023-07-14T17:42:48.313885+0000","last_fullsized":"2023-07-14T17:42:48.313885+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:17:07.258942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059529+0000","last_change":"2023-07-14T17:42:42.538553+0000","last_active":"2023-07-14T17:42:48.059529+0000","last_peered":"2023-07-14T17:42:48.059529+0000","last_clean":"2023-07-14T17:42:48.059529+0000","last_became_active":"2023-07-14T17:42:42.538251+0000","last_became_peered":"2023-07-14T17:42:42.538251+0000","last_unstale":"2023-07-14T17:42:48.059529+0000","last_undegraded":"2023-07-14T17:42:48.059529+0000","last_fullsized":"2023-07-14T17:42:48.059529+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:59:19.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123275+0000","last_change":"2023-07-14T17:42:42.543819+0000","last_active":"2023-07-14T17:42:48.123275+0000","last_peered":"2023-07-14T17:42:48.123275+0000","last_clean":"2023-07-14T17:42:48.123275+0000","last_became_active":"2023-07-14T17:42:42.543311+0000","last_became_peered":"2023-07-14T17:42:42.543311+0000","last_unstale":"2023-07-14T17:42:48.123275+0000","last_undegraded":"2023-07-14T17:42:48.123275+0000","last_fullsized":"2023-07-14T17:42:48.123275+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:43:18.425305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123144+0000","last_change":"2023-07-14T17:42:42.537525+0000","last_active":"2023-07-14T17:42:48.123144+0000","last_peered":"2023-07-14T17:42:48.123144+0000","last_clean":"2023-07-14T17:42:48.123144+0000","last_became_active":"2023-07-14T17:42:42.537242+0000","last_became_peered":"2023-07-14T17:42:42.537242+0000","last_unstale":"2023-07-14T17:42:48.123144+0000","last_undegraded":"2023-07-14T17:42:48.123144+0000","last_fullsized":"2023-07-14T17:42:48.123144+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:44:07.205354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313834+0000","last_change":"2023-07-14T17:42:42.547670+0000","last_active":"2023-07-14T17:42:48.313834+0000","last_peered":"2023-07-14T17:42:48.313834+0000","last_clean":"2023-07-14T17:42:48.313834+0000","last_became_active":"2023-07-14T17:42:42.547349+0000","last_became_peered":"2023-07-14T17:42:42.547349+0000","last_unstale":"2023-07-14T17:42:48.313834+0000","last_undegraded":"2023-07-14T17:42:48.313834+0000","last_fullsized":"2023-07-14T17:42:48.313834+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:11:40.630882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313964+0000","last_change":"2023-07-14T17:42:42.542741+0000","last_active":"2023-07-14T17:42:48.313964+0000","last_peered":"2023-07-14T17:42:48.313964+0000","last_clean":"2023-07-14T17:42:48.313964+0000","last_became_active":"2023-07-14T17:42:42.541972+0000","last_became_peered":"2023-07-14T17:42:42.541972+0000","last_unstale":"2023-07-14T17:42:48.313964+0000","last_undegraded":"2023-07-14T17:42:48.313964+0000","last_fullsized":"2023-07-14T17:42:48.313964+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:56:21.183121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059903+0000","last_change":"2023-07-14T17:42:42.535043+0000","last_active":"2023-07-14T17:42:48.059903+0000","last_peered":"2023-07-14T17:42:48.059903+0000","last_clean":"2023-07-14T17:42:48.059903+0000","last_became_active":"2023-07-14T17:42:42.534281+0000","last_became_peered":"2023-07-14T17:42:42.534281+0000","last_unstale":"2023-07-14T17:42:48.059903+0000","last_undegraded":"2023-07-14T17:42:48.059903+0000","last_fullsized":"2023-07-14T17:42:48.059903+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:59:12.865751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123808+0000","last_change":"2023-07-14T17:42:42.540866+0000","last_active":"2023-07-14T17:42:48.123808+0000","last_peered":"2023-07-14T17:42:48.123808+0000","last_clean":"2023-07-14T17:42:48.123808+0000","last_became_active":"2023-07-14T17:42:42.540568+0000","last_became_peered":"2023-07-14T17:42:42.540568+0000","last_unstale":"2023-07-14T17:42:48.123808+0000","last_undegraded":"2023-07-14T17:42:48.123808+0000","last_fullsized":"2023-07-14T17:42:48.123808+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:48:50.914640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540078+0000","last_change":"2023-07-14T17:42:42.541646+0000","last_active":"2023-07-14T17:42:47.540078+0000","last_peered":"2023-07-14T17:42:47.540078+0000","last_clean":"2023-07-14T17:42:47.540078+0000","last_became_active":"2023-07-14T17:42:42.541324+0000","last_became_peered":"2023-07-14T17:42:42.541324+0000","last_unstale":"2023-07-14T17:42:47.540078+0000","last_undegraded":"2023-07-14T17:42:47.540078+0000","last_fullsized":"2023-07-14T17:42:47.540078+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:11:55.891005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539881+0000","last_change":"2023-07-14T17:42:42.534650+0000","last_active":"2023-07-14T17:42:47.539881+0000","last_peered":"2023-07-14T17:42:47.539881+0000","last_clean":"2023-07-14T17:42:47.539881+0000","last_became_active":"2023-07-14T17:42:42.534351+0000","last_became_peered":"2023-07-14T17:42:42.534351+0000","last_unstale":"2023-07-14T17:42:47.539881+0000","last_undegraded":"2023-07-14T17:42:47.539881+0000","last_fullsized":"2023-07-14T17:42:47.539881+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:13:54.843952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539359+0000","last_change":"2023-07-14T17:42:42.535815+0000","last_active":"2023-07-14T17:42:47.539359+0000","last_peered":"2023-07-14T17:42:47.539359+0000","last_clean":"2023-07-14T17:42:47.539359+0000","last_became_active":"2023-07-14T17:42:42.535510+0000","last_became_peered":"2023-07-14T17:42:42.535510+0000","last_unstale":"2023-07-14T17:42:47.539359+0000","last_undegraded":"2023-07-14T17:42:47.539359+0000","last_fullsized":"2023-07-14T17:42:47.539359+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:39:09.947314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.315221+0000","last_change":"2023-07-14T17:42:42.551758+0000","last_active":"2023-07-14T17:42:48.315221+0000","last_peered":"2023-07-14T17:42:48.315221+0000","last_clean":"2023-07-14T17:42:48.315221+0000","last_became_active":"2023-07-14T17:42:42.551444+0000","last_became_peered":"2023-07-14T17:42:42.551444+0000","last_unstale":"2023-07-14T17:42:48.315221+0000","last_undegraded":"2023-07-14T17:42:48.315221+0000","last_fullsized":"2023-07-14T17:42:48.315221+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:42:02.798071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313340+0000","last_change":"2023-07-14T17:42:42.546355+0000","last_active":"2023-07-14T17:42:48.313340+0000","last_peered":"2023-07-14T17:42:48.313340+0000","last_clean":"2023-07-14T17:42:48.313340+0000","last_became_active":"2023-07-14T17:42:42.546079+0000","last_became_peered":"2023-07-14T17:42:42.546079+0000","last_unstale":"2023-07-14T17:42:48.313340+0000","last_undegraded":"2023-07-14T17:42:48.313340+0000","last_fullsized":"2023-07-14T17:42:48.313340+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:42:39.486035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059704+0000","last_change":"2023-07-14T17:42:42.543586+0000","last_active":"2023-07-14T17:42:48.059704+0000","last_peered":"2023-07-14T17:42:48.059704+0000","last_clean":"2023-07-14T17:42:48.059704+0000","last_became_active":"2023-07-14T17:42:42.543259+0000","last_became_peered":"2023-07-14T17:42:42.543259+0000","last_unstale":"2023-07-14T17:42:48.059704+0000","last_undegraded":"2023-07-14T17:42:48.059704+0000","last_fullsized":"2023-07-14T17:42:48.059704+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:24:35.934043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":43799,"num_objects":82,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":21,"num_read_kb":54,"num_write":188,"num_write_kb":100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":132419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":165,"ondisk_log_size":165,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481807,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":462514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]}]},{"osd":10,"up_from":69,"seq":296352743443,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28612,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716540,"statfs":{"total":95995035648,"available":95965736960,"internally_reserved":0,"allocated":1634304,"data_stored":1092893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.71699999999999997}]},{"osd":2,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.755},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.63600000000000001}]},{"osd":3,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.64600000000000002}]},{"osd":4,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.69099999999999995}]},{"osd":5,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.67800000000000005}]},{"osd":6,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.83099999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":8,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.873}]},{"osd":9,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.84599999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]}]},{"osd":9,"up_from":64,"seq":274877906967,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":462840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.29099999999999998}]},{"osd":1,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.375},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.42399999999999999}]},{"osd":2,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.45100000000000001}]},{"osd":3,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.311}]},{"osd":4,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.48999999999999999}]},{"osd":5,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.47399999999999998}]},{"osd":6,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.372},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.52500000000000002}]},{"osd":7,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.438}]},{"osd":8,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.53300000000000003}]},{"osd":10,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.46000000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":8,"up_from":57,"seq":244813135900,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":462670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53400000000000003,"15min":0.53300000000000003},"min":{"1min":0.35499999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.437}]},{"osd":1,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.372,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83899999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.40999999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.57999999999999996}]},{"osd":2,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45800000000000002,"15min":0.46100000000000002},"min":{"1min":0.27500000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60599999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.47199999999999998,"15min":0.47999999999999998},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56499999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.496}]},{"osd":3,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.36299999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.51900000000000002}]},{"osd":4,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85099999999999998,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64300000000000002,"15min":0.64200000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89700000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.66900000000000004}]},{"osd":5,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57399999999999995,"15min":0.58099999999999996},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72499999999999998,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.312,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.75,"5min":1.018,"15min":1.018},"last":0.89300000000000002}]},{"osd":6,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.66600000000000004,"15min":0.66800000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68500000000000005,"15min":0.69299999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84799999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.75600000000000001}]},{"osd":7,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.51000000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92400000000000004,"5min":1.008,"15min":1.008},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71099999999999997,"15min":0.70999999999999996},"min":{"1min":0.48899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88200000000000001,"5min":1.258,"15min":1.258},"last":0.82599999999999996}]},{"osd":9,"last update":"Fri Jul 14 17:42:19 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.76800000000000002}]},{"osd":10,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.80700000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]}]},{"osd":7,"up_from":49,"seq":210453397535,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28532,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716620,"statfs":{"total":95995035648,"available":95965818880,"internally_reserved":0,"allocated":1544192,"data_stored":1052882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.503},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.59499999999999997}]},{"osd":1,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.41499999999999998,"15min":0.42299999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.57799999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.46000000000000002,"15min":0.47099999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.51800000000000002,"5min":0.746,"15min":0.746},"last":0.68100000000000005}]},{"osd":2,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.628,"15min":0.60299999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":5.1050000000000004,"5min":5.1050000000000004,"15min":5.1050000000000004},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.71999999999999997,"5min":1.048,"15min":1.048},"last":0.83699999999999997}]},{"osd":3,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.436,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.59899999999999998,"15min":0.61199999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72499999999999998,"5min":0.995,"15min":0.995},"last":0.80600000000000005}]},{"osd":4,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.49299999999999999,"15min":0.499},"min":{"1min":0.312,"5min":0.31,"15min":0.31},"max":{"1min":0.59999999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.52000000000000002,"15min":0.53200000000000003},"min":{"1min":0.28899999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.66900000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.879}]},{"osd":5,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63400000000000001,"15min":0.63200000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64100000000000001,"15min":0.64300000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.94199999999999995,"5min":1.02,"15min":1.02},"last":0.96499999999999997}]},{"osd":6,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.64000000000000001,"15min":0.65000000000000002},"min":{"1min":0.435,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.94899999999999995}]},{"osd":8,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.45700000000000002,"5min":0.441,"15min":0.441},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.71299999999999997,"15min":0.72099999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0620000000000001,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":1.0620000000000001}]},{"osd":9,"last update":"Fri Jul 14 17:42:19 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":1.01},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.91200000000000003}]},{"osd":10,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":1.038},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.99099999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0249999999999999}]}]},{"osd":6,"up_from":44,"seq":188978561060,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":462606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:42:18 2023","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.36199999999999999,"15min":0.35299999999999998},"min":{"1min":0.29799999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.33100000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.35099999999999998}]},{"osd":1,"last update":"Fri Jul 14 17:42:18 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.57899999999999996,"15min":0.56699999999999995},"min":{"1min":0.45400000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.56999999999999995,"15min":0.56299999999999994},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.58099999999999996}]},{"osd":2,"last update":"Fri Jul 14 17:42:18 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56999999999999995,"15min":0.56799999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.76700000000000002,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.59799999999999998,"15min":0.59199999999999997},"min":{"1min":0.438,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.53700000000000003}]},{"osd":3,"last update":"Fri Jul 14 17:42:18 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.45600000000000002,"15min":0.45100000000000001},"min":{"1min":0.32200000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69299999999999995,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.46899999999999997,"15min":0.45300000000000001},"min":{"1min":0.34599999999999997,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.40200000000000002}]},{"osd":4,"last update":"Fri Jul 14 17:42:18 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.59499999999999997,"15min":0.59099999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.60999999999999999,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.58999999999999997}]},{"osd":5,"last update":"Fri Jul 14 17:42:18 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.59999999999999998,"15min":0.59299999999999997},"min":{"1min":0.44400000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.66200000000000003,"15min":0.65500000000000003},"min":{"1min":0.45800000000000002,"5min":0.443,"15min":0.443},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.59899999999999998}]},{"osd":7,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.66600000000000004,"15min":0.66000000000000003},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.498},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.68999999999999995,"15min":0.68799999999999994},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.60799999999999998}]},{"osd":8,"last update":"Fri Jul 14 17:43:04 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.73599999999999999,"15min":0.755},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.98999999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.69499999999999995,"15min":0.71099999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92500000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.50600000000000001}]},{"osd":9,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.46200000000000002}]},{"osd":10,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.63300000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":484811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:42:37 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.48899999999999999,"15min":0.46200000000000002},"min":{"1min":0.38,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.82999999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.436},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.50600000000000001,"15min":0.46899999999999997},"min":{"1min":0.34499999999999997,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.40899999999999997}]},{"osd":2,"last update":"Fri Jul 14 17:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.54200000000000004,"15min":0.497},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.86099999999999999,"5min":0.879,"15min":0.879},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.52300000000000002,"15min":0.48899999999999999},"min":{"1min":0.36499999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.73299999999999998}]},{"osd":3,"last update":"Fri Jul 14 17:42:18 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.52900000000000003,"15min":0.51400000000000001},"min":{"1min":0.33400000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.81200000000000006,"5min":0.873,"15min":0.873},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.56499999999999995,"15min":0.54100000000000004},"min":{"1min":0.443,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.56299999999999994}]},{"osd":4,"last update":"Fri Jul 14 17:42:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.42799999999999999,"15min":0.42199999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66300000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.44,"15min":0.42799999999999999},"min":{"1min":0.29999999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.47199999999999998}]},{"osd":5,"last update":"Fri Jul 14 17:42:59 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.60899999999999999,"15min":0.58499999999999996},"min":{"1min":0.48299999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.58199999999999996,"15min":0.57099999999999995},"min":{"1min":0.42699999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.69699999999999995}]},{"osd":6,"last update":"Fri Jul 14 17:42:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.40899999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60099999999999998,"15min":0.59899999999999998},"min":{"1min":0.41899999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.82299999999999995,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.60099999999999998}]},{"osd":7,"last update":"Fri Jul 14 17:42:39 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.53000000000000003,"15min":0.51500000000000001},"min":{"1min":0.378,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86499999999999999,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52700000000000002,"15min":0.52400000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.64600000000000002}]},{"osd":8,"last update":"Fri Jul 14 17:42:59 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.65900000000000003,"15min":0.65600000000000003},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1,"5min":1,"15min":1},"last":0.747},{"interface":"front","average":{"1min":0.746,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.54000000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.81799999999999995}]},{"osd":9,"last update":"Fri Jul 14 17:42:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.625},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.68200000000000005}]},{"osd":10,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.78100000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":463950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 17:42:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.51700000000000002,"15min":0.505},"min":{"1min":0.503,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.54100000000000004,"15min":0.52100000000000002},"min":{"1min":0.45000000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.46200000000000002}]},{"osd":2,"last update":"Fri Jul 14 17:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55900000000000005,"15min":0.53400000000000003},"min":{"1min":0.40999999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56999999999999995,"15min":0.55300000000000005},"min":{"1min":0.379,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.441}]},{"osd":3,"last update":"Fri Jul 14 17:41:15 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44700000000000001,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.40999999999999998,"15min":0.40600000000000003},"min":{"1min":0.27900000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.61699999999999999,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.47699999999999998}]},{"osd":4,"last update":"Fri Jul 14 17:42:37 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.55400000000000005,"15min":0.54000000000000004},"min":{"1min":0.435,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.40699999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.85099999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.59799999999999998}]},{"osd":5,"last update":"Fri Jul 14 17:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56499999999999995,"15min":0.55100000000000005},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.755,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59599999999999997,"15min":0.58999999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84099999999999997,"5min":0.88,"15min":0.88},"last":0.499}]},{"osd":6,"last update":"Fri Jul 14 17:42:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.48599999999999999,"15min":0.47099999999999997},"min":{"1min":0.33200000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.501,"15min":0.48799999999999999},"min":{"1min":0.34300000000000003,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.53200000000000003}]},{"osd":7,"last update":"Fri Jul 14 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.66900000000000004}]},{"osd":8,"last update":"Fri Jul 14 17:42:02 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.58699999999999997}]},{"osd":9,"last update":"Fri Jul 14 17:42:19 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.64100000000000001}]},{"osd":10,"last update":"Fri Jul 14 17:42:44 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.624},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.63300000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":502440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.47499999999999998},"min":{"1min":0.41099999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.5},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.48299999999999998,"15min":0.46700000000000003},"min":{"1min":0.39100000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.53000000000000003}]},{"osd":1,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.53100000000000003,"15min":0.48299999999999998},"min":{"1min":0.35199999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.55300000000000005,"15min":0.50900000000000001},"min":{"1min":0.5,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.80300000000000005}]},{"osd":3,"last update":"Fri Jul 14 17:42:22 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.55700000000000005,"15min":0.54100000000000004},"min":{"1min":0.47099999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.54900000000000004,"15min":0.53600000000000003},"min":{"1min":0.436,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.59499999999999997}]},{"osd":4,"last update":"Fri Jul 14 17:41:35 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.51900000000000002,"15min":0.51300000000000001},"min":{"1min":0.38400000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.55400000000000005,"15min":0.54600000000000004},"min":{"1min":0.39900000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.86599999999999999}]},{"osd":5,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.45600000000000002,"15min":0.43099999999999999},"min":{"1min":0.30399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.45200000000000001,"15min":0.437},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.56200000000000006}]},{"osd":6,"last update":"Fri Jul 14 17:42:22 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.58799999999999997,"15min":0.57599999999999996},"min":{"1min":0.48299999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.70999999999999996}]},{"osd":7,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.59099999999999997,"15min":0.57499999999999996},"min":{"1min":0.40200000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.66000000000000003,"15min":0.64600000000000002},"min":{"1min":0.52500000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.94499999999999995}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.61299999999999999,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.59299999999999997,"15min":0.57399999999999995},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.69399999999999995}]},{"osd":9,"last update":"Fri Jul 14 17:42:22 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.91800000000000004}]},{"osd":10,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.876},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.90600000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182449,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":462340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.33700000000000002,"15min":0.32100000000000001},"min":{"1min":0.28899999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.504,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.307},{"interface":"front","average":{"1min":0.437,"5min":0.35099999999999998,"15min":0.32900000000000001},"min":{"1min":0.29499999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.33300000000000002}]},{"osd":1,"last update":"Fri Jul 14 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.52600000000000002,"15min":0.5},"min":{"1min":0.41299999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.80600000000000005,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.53100000000000003,"15min":0.501},"min":{"1min":0.47099999999999997,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.84099999999999997,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.501}]},{"osd":2,"last update":"Fri Jul 14 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.54200000000000004,"15min":0.52200000000000002},"min":{"1min":0.42599999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.55800000000000005,"15min":0.52600000000000002},"min":{"1min":0.436,"5min":0.379,"15min":0.379},"max":{"1min":0.91900000000000004,"5min":1.423,"15min":1.423},"last":0.60999999999999999}]},{"osd":4,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58099999999999996,"15min":0.56699999999999995},"min":{"1min":0.37,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71299999999999997,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60299999999999998,"15min":0.58199999999999996},"min":{"1min":0.50600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84199999999999997,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.48999999999999999}]},{"osd":5,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.38100000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85299999999999998,"5min":1.593,"15min":1.593},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.45400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.85699999999999998,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.621}]},{"osd":6,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.53700000000000003,"15min":0.53300000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.78700000000000003,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.48199999999999998,"15min":0.47999999999999998},"min":{"1min":0.34999999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67500000000000004,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.42999999999999999}]},{"osd":7,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.63700000000000001,"15min":0.64300000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79300000000000004,"5min":1.28,"15min":1.28},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.65200000000000002,"15min":0.64400000000000002},"min":{"1min":0.56200000000000006,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.89000000000000001,"5min":1.03,"15min":1.03},"last":0.56999999999999995}]},{"osd":8,"last update":"Fri Jul 14 17:41:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.55800000000000005}]},{"osd":9,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.41699999999999998}]},{"osd":10,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.59999999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953516,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":462528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.35399999999999998,"5min":0.311,"15min":0.311},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.52100000000000002,"15min":0.502},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.496}]},{"osd":1,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.39700000000000002,"15min":0.375},"min":{"1min":0.30499999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.41099999999999998,"15min":0.39500000000000002},"min":{"1min":0.29799999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.67200000000000004,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.439}]},{"osd":2,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.42199999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.59399999999999997,"15min":0.57199999999999995},"min":{"1min":0.437,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85399999999999998,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.73399999999999999}]},{"osd":3,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.57899999999999996,"15min":0.54500000000000004},"min":{"1min":0.39600000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.96599999999999997,"5min":1.498,"15min":1.498},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55400000000000005,"15min":0.53200000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.84399999999999997,"5min":1.629,"15min":1.629},"last":0.57099999999999995}]},{"osd":5,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.63100000000000001,"15min":0.61099999999999999},"min":{"1min":0.51500000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89800000000000002,"5min":1.256,"15min":1.256},"last":0.755},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.624,"15min":0.60999999999999999},"min":{"1min":0.44900000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.68400000000000005}]},{"osd":6,"last update":"Fri Jul 14 17:42:18 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.745,"15min":0.753},"min":{"1min":0.51800000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91400000000000003,"5min":2.7789999999999999,"15min":2.7789999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.67900000000000005,"15min":0.66600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.2210000000000001,"5min":1.476,"15min":1.476},"last":0.55500000000000005}]},{"osd":7,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.55000000000000004,"15min":0.52300000000000002},"min":{"1min":0.39000000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.59399999999999997,"15min":0.57799999999999996},"min":{"1min":0.38100000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.63500000000000001}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.65600000000000003,"15min":0.64400000000000002},"min":{"1min":0.41999999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72299999999999998,"15min":0.71399999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.66100000000000003}]},{"osd":9,"last update":"Fri Jul 14 17:42:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.70699999999999996}]},{"osd":10,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.876}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90700000000000003}]}]},{"osd":5,"up_from":37,"seq":158913789993,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1589248,"data_stored":1059558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47099999999999997,"15min":0.44700000000000001},"min":{"1min":0.35499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77800000000000002,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.48299999999999998,"15min":0.45700000000000002},"min":{"1min":0.35599999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65400000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.52400000000000002}]},{"osd":1,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.55100000000000005,"15min":0.497},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":5.3630000000000004,"5min":5.3630000000000004,"15min":5.3630000000000004},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51300000000000001,"15min":0.48999999999999999},"min":{"1min":0.42399999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.82299999999999995}]},{"osd":2,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.33500000000000002},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.376,"15min":0.35999999999999999},"min":{"1min":0.30499999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.59499999999999997}]},{"osd":3,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.501,"15min":0.495},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.51800000000000002,"15min":0.495},"min":{"1min":0.38700000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.747,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.64500000000000002}]},{"osd":4,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.53500000000000003,"15min":0.50800000000000001},"min":{"1min":0.434,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78100000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54300000000000004,"15min":0.52000000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.72499999999999998}]},{"osd":6,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.78700000000000003,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.54600000000000004,"15min":0.53600000000000003},"min":{"1min":0.45400000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78800000000000003,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.68300000000000005}]},{"osd":7,"last update":"Fri Jul 14 17:42:37 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57099999999999995,"15min":0.56799999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.60599999999999998,"15min":0.60399999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.79900000000000004,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.77500000000000002}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.54700000000000004,"15min":0.54000000000000004},"min":{"1min":0.38100000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.754,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.372,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.876,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.66900000000000004}]},{"osd":9,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.86899999999999999}]},{"osd":10,"last update":"Fri Jul 14 17:42:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":5.3330000000000002,"5min":5.3330000000000002,"15min":5.3330000000000002},"last":0.83999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":40219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":40781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":40593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T17:43:06.398 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-07-14T17:43:06.399 INFO:tasks.cephadm:Setting up client nodes... 2023-07-14T17:43:06.399 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-07-14T17:43:06.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:05 smithi178 ceph-mon[129368]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:43:06.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:05 smithi178 ceph-mon[129368]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:43:06.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:05 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3425644091' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:43:06.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:05 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3004495923' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:43:06.985 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:07.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:06 smithi088 ceph-mon[127465]: pgmap v267: 129 pgs: 129 active+clean; 619 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7.2 KiB/s wr, 13 op/s 2023-07-14T17:43:07.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:06 smithi088 ceph-mon[127465]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:43:07.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:06 smithi107 ceph-mon[128999]: pgmap v267: 129 pgs: 129 active+clean; 619 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7.2 KiB/s wr, 13 op/s 2023-07-14T17:43:07.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:06 smithi107 ceph-mon[128999]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:43:07.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:06 smithi178 ceph-mon[129368]: pgmap v267: 129 pgs: 129 active+clean; 619 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 7.2 KiB/s wr, 13 op/s 2023-07-14T17:43:07.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:06 smithi178 ceph-mon[129368]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:43:08.077 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:07 smithi088 ceph-mon[127465]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:43:08.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:07 smithi107 ceph-mon[128999]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:43:08.224 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:43:07 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-b[154079]: 2023-07-14T17:43:07.891+0000 7fb592e7d700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-07-14T17:43:08.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.282+0000 7fe517d41700 1 -- 172.21.15.88:0/4170524622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5100ff860 msgr2=0x7fe510107130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:08.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.282+0000 7fe517d41700 1 --2- 172.21.15.88:0/4170524622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5100ff860 0x7fe510107130 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe50800b1d0 tx=0x7fe50800b4e0 comp rx=0 tx=0).stop 2023-07-14T17:43:08.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.282+0000 7fe517d41700 1 -- 172.21.15.88:0/4170524622 shutdown_connections 2023-07-14T17:43:08.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.282+0000 7fe517d41700 1 --2- 172.21.15.88:0/4170524622 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5100ff860 0x7fe510107130 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:08.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.282+0000 7fe517d41700 1 --2- 172.21.15.88:0/4170524622 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe510100190 0x7fe5101005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:08.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.282+0000 7fe517d41700 1 --2- 172.21.15.88:0/4170524622 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5100feed0 0x7fe5100ff320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:08.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.282+0000 7fe517d41700 1 -- 172.21.15.88:0/4170524622 >> 172.21.15.88:0/4170524622 conn(0x7fe5100f99b0 msgr2=0x7fe5100fbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:08.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.282+0000 7fe517d41700 1 -- 172.21.15.88:0/4170524622 shutdown_connections 2023-07-14T17:43:08.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.282+0000 7fe517d41700 1 -- 172.21.15.88:0/4170524622 wait complete. 2023-07-14T17:43:08.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe517d41700 1 Processor -- start 2023-07-14T17:43:08.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe517d41700 1 -- start start 2023-07-14T17:43:08.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe517d41700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5100feed0 0x7fe510197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:08.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe517d41700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5100ff860 0x7fe510197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:08.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe517d41700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe510100190 0x7fe51019efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:08.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe517d41700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51019f5a0 con 0x7fe5100feed0 2023-07-14T17:43:08.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe517d41700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51019f6e0 con 0x7fe5100ff860 2023-07-14T17:43:08.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe517d41700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51019f870 con 0x7fe510100190 2023-07-14T17:43:08.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe515add700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5100feed0 0x7fe510197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:08.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe515add700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5100feed0 0x7fe510197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42048/0 (socket says 172.21.15.88:42048) 2023-07-14T17:43:08.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe515add700 1 -- 172.21.15.88:0/991736292 learned_addr learned my addr 172.21.15.88:0/991736292 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:08.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe5162de700 1 --2- 172.21.15.88:0/991736292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe510100190 0x7fe51019efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:08.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe5152dc700 1 --2- 172.21.15.88:0/991736292 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5100ff860 0x7fe510197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:08.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe515add700 1 -- 172.21.15.88:0/991736292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe510100190 msgr2=0x7fe51019efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:08.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe515add700 1 --2- 172.21.15.88:0/991736292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe510100190 0x7fe51019efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:08.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe515add700 1 -- 172.21.15.88:0/991736292 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5100ff860 msgr2=0x7fe510197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:08.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe515add700 1 --2- 172.21.15.88:0/991736292 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5100ff860 0x7fe510197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:08.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe515add700 1 -- 172.21.15.88:0/991736292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe508009d00 con 0x7fe5100feed0 2023-07-14T17:43:08.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.283+0000 7fe515add700 1 --2- 172.21.15.88:0/991736292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5100feed0 0x7fe510197a10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe50000c9c0 tx=0x7fe50000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:08.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.284+0000 7fe506ffd700 1 -- 172.21.15.88:0/991736292 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe500007ac0 con 0x7fe5100feed0 2023-07-14T17:43:08.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.284+0000 7fe517d41700 1 -- 172.21.15.88:0/991736292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe51019fb50 con 0x7fe5100feed0 2023-07-14T17:43:08.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.284+0000 7fe506ffd700 1 -- 172.21.15.88:0/991736292 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe500007c20 con 0x7fe5100feed0 2023-07-14T17:43:08.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.284+0000 7fe506ffd700 1 -- 172.21.15.88:0/991736292 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe500018660 con 0x7fe5100feed0 2023-07-14T17:43:08.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.284+0000 7fe517d41700 1 -- 172.21.15.88:0/991736292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5101a00a0 con 0x7fe5100feed0 2023-07-14T17:43:08.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.285+0000 7fe517d41700 1 -- 172.21.15.88:0/991736292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe510109210 con 0x7fe5100feed0 2023-07-14T17:43:08.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.286+0000 7fe506ffd700 1 -- 172.21.15.88:0/991736292 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe50000f450 con 0x7fe5100feed0 2023-07-14T17:43:08.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.286+0000 7fe506ffd700 1 --2- 172.21.15.88:0/991736292 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe4fc0a11e0 0x7fe4fc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:08.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.286+0000 7fe5152dc700 1 --2- 172.21.15.88:0/991736292 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe4fc0a11e0 0x7fe4fc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:08.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.286+0000 7fe506ffd700 1 -- 172.21.15.88:0/991736292 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe5000ccdd0 con 0x7fe5100feed0 2023-07-14T17:43:08.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.286+0000 7fe5152dc700 1 --2- 172.21.15.88:0/991736292 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe4fc0a11e0 0x7fe4fc0a3690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe50c005950 tx=0x7fe50c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:08.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.289+0000 7fe506ffd700 1 -- 172.21.15.88:0/991736292 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe500096260 con 0x7fe5100feed0 2023-07-14T17:43:08.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:07 smithi178 ceph-mon[129368]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:43:08.530 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.548+0000 7fe517d41700 1 -- 172.21.15.88:0/991736292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fe510105b00 con 0x7fe5100feed0 2023-07-14T17:43:08.534 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.552+0000 7fe506ffd700 1 -- 172.21.15.88:0/991736292 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7fe50000fe60 con 0x7fe5100feed0 2023-07-14T17:43:08.535 INFO:teuthology.orchestra.run.smithi088.stdout:[client.0] 2023-07-14T17:43:08.535 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAsibFk5Eu+IBAAxNteff0Vg6bB1twA3mRyaw== 2023-07-14T17:43:08.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.554+0000 7fe517d41700 1 -- 172.21.15.88:0/991736292 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe4fc0a11e0 msgr2=0x7fe4fc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:08.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.554+0000 7fe517d41700 1 --2- 172.21.15.88:0/991736292 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe4fc0a11e0 0x7fe4fc0a3690 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe50c005950 tx=0x7fe50c0058e0 comp rx=0 tx=0).stop 2023-07-14T17:43:08.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.554+0000 7fe517d41700 1 -- 172.21.15.88:0/991736292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5100feed0 msgr2=0x7fe510197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:08.537 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.554+0000 7fe517d41700 1 --2- 172.21.15.88:0/991736292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5100feed0 0x7fe510197a10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe50000c9c0 tx=0x7fe50000cd80 comp rx=0 tx=0).stop 2023-07-14T17:43:08.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.554+0000 7fe517d41700 1 -- 172.21.15.88:0/991736292 shutdown_connections 2023-07-14T17:43:08.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.554+0000 7fe517d41700 1 --2- 172.21.15.88:0/991736292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe510100190 0x7fe51019efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:08.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.554+0000 7fe517d41700 1 --2- 172.21.15.88:0/991736292 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe4fc0a11e0 0x7fe4fc0a3690 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:08.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.554+0000 7fe517d41700 1 --2- 172.21.15.88:0/991736292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5100feed0 0x7fe510197a10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:08.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.554+0000 7fe517d41700 1 --2- 172.21.15.88:0/991736292 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5100ff860 0x7fe510197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:08.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.554+0000 7fe517d41700 1 -- 172.21.15.88:0/991736292 >> 172.21.15.88:0/991736292 conn(0x7fe5100f99b0 msgr2=0x7fe5100fbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:08.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.555+0000 7fe517d41700 1 -- 172.21.15.88:0/991736292 shutdown_connections 2023-07-14T17:43:08.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:08.555+0000 7fe517d41700 1 -- 172.21.15.88:0/991736292 wait complete. 2023-07-14T17:43:08.933 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:43:08.934 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-07-14T17:43:08.934 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-07-14T17:43:08.975 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-07-14T17:43:08.976 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-07-14T17:43:08.976 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph mgr dump --format=json 2023-07-14T17:43:09.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:08 smithi088 ceph-mon[127465]: pgmap v268: 129 pgs: 129 active+clean; 620 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7.1 KiB/s wr, 14 op/s 2023-07-14T17:43:09.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:08 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/991736292' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T17:43:09.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:08 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/991736292' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T17:43:09.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:08 smithi107 ceph-mon[128999]: pgmap v268: 129 pgs: 129 active+clean; 620 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7.1 KiB/s wr, 14 op/s 2023-07-14T17:43:09.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:08 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/991736292' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T17:43:09.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:08 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/991736292' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T17:43:09.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:08 smithi178 ceph-mon[129368]: pgmap v268: 129 pgs: 129 active+clean; 620 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7.1 KiB/s wr, 14 op/s 2023-07-14T17:43:09.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:08 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/991736292' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T17:43:09.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:08 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/991736292' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T17:43:09.565 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:10.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.831+0000 7fd00a7fd700 1 -- 172.21.15.88:0/3082782273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0040ff860 msgr2=0x7fd004103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:10.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.831+0000 7fd00a7fd700 1 --2- 172.21.15.88:0/3082782273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0040ff860 0x7fd004103830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fcff80099c0 tx=0x7fcff8009cd0 comp rx=0 tx=0).stop 2023-07-14T17:43:10.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.831+0000 7fd00a7fd700 1 -- 172.21.15.88:0/3082782273 shutdown_connections 2023-07-14T17:43:10.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.831+0000 7fd00a7fd700 1 --2- 172.21.15.88:0/3082782273 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd004103f60 0x7fd00410bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:10.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.831+0000 7fd00a7fd700 1 --2- 172.21.15.88:0/3082782273 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0040feec0 0x7fd0040ff290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:10.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.831+0000 7fd00a7fd700 1 --2- 172.21.15.88:0/3082782273 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0040ff860 0x7fd004103830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:10.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.831+0000 7fd00a7fd700 1 -- 172.21.15.88:0/3082782273 >> 172.21.15.88:0/3082782273 conn(0x7fd0040fa720 msgr2=0x7fd0040fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:10.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.831+0000 7fd00a7fd700 1 -- 172.21.15.88:0/3082782273 shutdown_connections 2023-07-14T17:43:10.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.831+0000 7fd00a7fd700 1 -- 172.21.15.88:0/3082782273 wait complete. 2023-07-14T17:43:10.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.832+0000 7fd00a7fd700 1 Processor -- start 2023-07-14T17:43:10.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.832+0000 7fd00a7fd700 1 -- start start 2023-07-14T17:43:10.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.832+0000 7fd00a7fd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0040feec0 0x7fd0041948f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:10.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.832+0000 7fd00a7fd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0040ff860 0x7fd004194e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:10.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.832+0000 7fd00a7fd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd004103f60 0x7fd004195370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:10.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.832+0000 7fd00a7fd700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0041a4130 con 0x7fd0040ff860 2023-07-14T17:43:10.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.832+0000 7fd00a7fd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0041a42a0 con 0x7fd004103f60 2023-07-14T17:43:10.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.832+0000 7fd00a7fd700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0041a4460 con 0x7fd0040feec0 2023-07-14T17:43:10.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.833+0000 7fd0037fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0040ff860 0x7fd004194e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:10.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.833+0000 7fd0037fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0040ff860 0x7fd004194e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42060/0 (socket says 172.21.15.88:42060) 2023-07-14T17:43:10.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.833+0000 7fd0037fe700 1 -- 172.21.15.88:0/2439737368 learned_addr learned my addr 172.21.15.88:0/2439737368 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:10.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.833+0000 7fd003fff700 1 --2- 172.21.15.88:0/2439737368 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0040feec0 0x7fd0041948f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:10.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.833+0000 7fd008d9a700 1 --2- 172.21.15.88:0/2439737368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd004103f60 0x7fd004195370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:10.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.833+0000 7fd0037fe700 1 -- 172.21.15.88:0/2439737368 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0040feec0 msgr2=0x7fd0041948f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:10.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.833+0000 7fd0037fe700 1 --2- 172.21.15.88:0/2439737368 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0040feec0 0x7fd0041948f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:10.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.833+0000 7fd0037fe700 1 -- 172.21.15.88:0/2439737368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd004103f60 msgr2=0x7fd004195370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:10.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.833+0000 7fd0037fe700 1 --2- 172.21.15.88:0/2439737368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd004103f60 0x7fd004195370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:10.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.833+0000 7fd0037fe700 1 -- 172.21.15.88:0/2439737368 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff80095f0 con 0x7fd0040ff860 2023-07-14T17:43:10.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.833+0000 7fd0037fe700 1 --2- 172.21.15.88:0/2439737368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0040ff860 0x7fd004194e30 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcff8006010 tx=0x7fcff8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:10.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.833+0000 7fd0017fa700 1 -- 172.21.15.88:0/2439737368 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff801d070 con 0x7fd0040ff860 2023-07-14T17:43:10.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.833+0000 7fd00a7fd700 1 -- 172.21.15.88:0/2439737368 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0041a46e0 con 0x7fd0040ff860 2023-07-14T17:43:10.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.833+0000 7fd0017fa700 1 -- 172.21.15.88:0/2439737368 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcff8004c70 con 0x7fd0040ff860 2023-07-14T17:43:10.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.833+0000 7fd0017fa700 1 -- 172.21.15.88:0/2439737368 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff8003d50 con 0x7fd0040ff860 2023-07-14T17:43:10.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.833+0000 7fd00a7fd700 1 -- 172.21.15.88:0/2439737368 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0041a4af0 con 0x7fd0040ff860 2023-07-14T17:43:10.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.834+0000 7fd00a7fd700 1 -- 172.21.15.88:0/2439737368 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd00404e340 con 0x7fd0040ff860 2023-07-14T17:43:10.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.835+0000 7fd0017fa700 1 -- 172.21.15.88:0/2439737368 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fcff8004de0 con 0x7fd0040ff860 2023-07-14T17:43:10.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.836+0000 7fd0017fa700 1 --2- 172.21.15.88:0/2439737368 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcfec0a11e0 0x7fcfec0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:10.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.836+0000 7fd0017fa700 1 -- 172.21.15.88:0/2439737368 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fcff80cde50 con 0x7fd0040ff860 2023-07-14T17:43:10.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.836+0000 7fd003fff700 1 --2- 172.21.15.88:0/2439737368 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcfec0a11e0 0x7fcfec0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:10.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.836+0000 7fd003fff700 1 --2- 172.21.15.88:0/2439737368 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcfec0a11e0 0x7fcfec0a3690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcff4005950 tx=0x7fcff400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:10.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:10.839+0000 7fd0017fa700 1 -- 172.21.15.88:0/2439737368 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcff8097280 con 0x7fd0040ff860 2023-07-14T17:43:11.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:11.082+0000 7fd00a7fd700 1 -- 172.21.15.88:0/2439737368 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fd0041a4d10 con 0x7fd0040ff860 2023-07-14T17:43:11.069 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:11.087+0000 7fd0017fa700 1 -- 172.21.15.88:0/2439737368 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260125 (secure 0 0 0) 0x7fcff809a8a0 con 0x7fd0040ff860 2023-07-14T17:43:11.072 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:43:11.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:11.090+0000 7fd00a7fd700 1 -- 172.21.15.88:0/2439737368 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcfec0a11e0 msgr2=0x7fcfec0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:11.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:11.090+0000 7fd00a7fd700 1 --2- 172.21.15.88:0/2439737368 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcfec0a11e0 0x7fcfec0a3690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcff4005950 tx=0x7fcff400a300 comp rx=0 tx=0).stop 2023-07-14T17:43:11.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:11.090+0000 7fd00a7fd700 1 -- 172.21.15.88:0/2439737368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0040ff860 msgr2=0x7fd004194e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:11.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:11.090+0000 7fd00a7fd700 1 --2- 172.21.15.88:0/2439737368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0040ff860 0x7fd004194e30 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fcff8006010 tx=0x7fcff8004730 comp rx=0 tx=0).stop 2023-07-14T17:43:11.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:11.090+0000 7fd00a7fd700 1 -- 172.21.15.88:0/2439737368 shutdown_connections 2023-07-14T17:43:11.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:11.090+0000 7fd00a7fd700 1 --2- 172.21.15.88:0/2439737368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd004103f60 0x7fd004195370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:11.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:11.090+0000 7fd00a7fd700 1 --2- 172.21.15.88:0/2439737368 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd0040feec0 0x7fd0041948f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:11.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:11.090+0000 7fd00a7fd700 1 --2- 172.21.15.88:0/2439737368 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcfec0a11e0 0x7fcfec0a3690 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:11.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:11.090+0000 7fd00a7fd700 1 --2- 172.21.15.88:0/2439737368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0040ff860 0x7fd004194e30 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:11.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:11.090+0000 7fd00a7fd700 1 -- 172.21.15.88:0/2439737368 >> 172.21.15.88:0/2439737368 conn(0x7fd0040fa720 msgr2=0x7fd0040fb380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:11.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:11.090+0000 7fd00a7fd700 1 -- 172.21.15.88:0/2439737368 shutdown_connections 2023-07-14T17:43:11.075 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:11.090+0000 7fd00a7fd700 1 -- 172.21.15.88:0/2439737368 wait complete. 2023-07-14T17:43:11.220 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:10 smithi088 ceph-mon[127465]: pgmap v269: 129 pgs: 129 active+clean; 621 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 7.7 KiB/s wr, 18 op/s 2023-07-14T17:43:11.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:10 smithi107 ceph-mon[128999]: pgmap v269: 129 pgs: 129 active+clean; 621 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 7.7 KiB/s wr, 18 op/s 2023-07-14T17:43:11.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:10 smithi178 ceph-mon[129368]: pgmap v269: 129 pgs: 129 active+clean; 621 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 7.7 KiB/s wr, 18 op/s 2023-07-14T17:43:11.423 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":17,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":4262759610},{"type":"v1","addr":"172.21.15.88:6801","nonce":4262759610}]},"active_addr":"172.21.15.88:6801/4262759610","active_change":"2023-07-14T17:36:43.799847+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.88:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":4012505202}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":3955688508}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":1173263538}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":2620671934}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.88:0","nonce":463975183}]}]} 2023-07-14T17:43:11.430 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-07-14T17:43:11.430 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-07-14T17:43:11.431 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T17:43:11.981 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:12.220 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:11 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2439737368' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T17:43:12.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:11 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2439737368' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T17:43:12.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:11 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2439737368' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T17:43:13.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:12 smithi107 ceph-mon[128999]: pgmap v270: 129 pgs: 129 active+clean; 626 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.1 KiB/s wr, 22 op/s 2023-07-14T17:43:13.250 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:12 smithi088 ceph-mon[127465]: pgmap v270: 129 pgs: 129 active+clean; 626 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.1 KiB/s wr, 22 op/s 2023-07-14T17:43:13.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:12 smithi178 ceph-mon[129368]: pgmap v270: 129 pgs: 129 active+clean; 626 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.1 KiB/s wr, 22 op/s 2023-07-14T17:43:13.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.499+0000 7fdbca26f700 1 -- 172.21.15.88:0/4072785905 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbc4103f10 msgr2=0x7fdbc410ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:13.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.499+0000 7fdbca26f700 1 --2- 172.21.15.88:0/4072785905 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbc4103f10 0x7fdbc410ba90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc00b0d0 tx=0x7fdbbc00b3e0 comp rx=0 tx=0).stop 2023-07-14T17:43:13.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.499+0000 7fdbca26f700 1 -- 172.21.15.88:0/4072785905 shutdown_connections 2023-07-14T17:43:13.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.499+0000 7fdbca26f700 1 --2- 172.21.15.88:0/4072785905 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbc4103f10 0x7fdbc410ba90 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:13.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.499+0000 7fdbca26f700 1 --2- 172.21.15.88:0/4072785905 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbc40fee70 0x7fdbc40ff240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:13.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.499+0000 7fdbca26f700 1 --2- 172.21.15.88:0/4072785905 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbc40ff810 0x7fdbc41037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:13.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.499+0000 7fdbca26f700 1 -- 172.21.15.88:0/4072785905 >> 172.21.15.88:0/4072785905 conn(0x7fdbc4075ae0 msgr2=0x7fdbc4075ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:13.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.499+0000 7fdbca26f700 1 -- 172.21.15.88:0/4072785905 shutdown_connections 2023-07-14T17:43:13.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.499+0000 7fdbca26f700 1 -- 172.21.15.88:0/4072785905 wait complete. 2023-07-14T17:43:13.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.500+0000 7fdbca26f700 1 Processor -- start 2023-07-14T17:43:13.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.500+0000 7fdbca26f700 1 -- start start 2023-07-14T17:43:13.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.500+0000 7fdbca26f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbc40fee70 0x7fdbc4194ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:13.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.500+0000 7fdbca26f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbc40ff810 0x7fdbc41951e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:13.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbca26f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbc4103f10 0x7fdbc41985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:13.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbca26f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc4198ba0 con 0x7fdbc40fee70 2023-07-14T17:43:13.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbca26f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc4198d10 con 0x7fdbc40ff810 2023-07-14T17:43:13.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbca26f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc4198ed0 con 0x7fdbc4103f10 2023-07-14T17:43:13.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbc3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbc40fee70 0x7fdbc4194ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:13.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbc3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbc40fee70 0x7fdbc4194ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42070/0 (socket says 172.21.15.88:42070) 2023-07-14T17:43:13.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbc3fff700 1 -- 172.21.15.88:0/4057090181 learned_addr learned my addr 172.21.15.88:0/4057090181 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:13.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbc37fe700 1 --2- 172.21.15.88:0/4057090181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbc40ff810 0x7fdbc41951e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:13.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbc880c700 1 --2- 172.21.15.88:0/4057090181 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbc4103f10 0x7fdbc41985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:13.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbc3fff700 1 -- 172.21.15.88:0/4057090181 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbc4103f10 msgr2=0x7fdbc41985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:13.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbc3fff700 1 --2- 172.21.15.88:0/4057090181 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbc4103f10 0x7fdbc41985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:13.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbc3fff700 1 -- 172.21.15.88:0/4057090181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbc40ff810 msgr2=0x7fdbc41951e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:13.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbc3fff700 1 --2- 172.21.15.88:0/4057090181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbc40ff810 0x7fdbc41951e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:13.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbc3fff700 1 -- 172.21.15.88:0/4057090181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbbc009d00 con 0x7fdbc40fee70 2023-07-14T17:43:13.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbc3fff700 1 --2- 172.21.15.88:0/4057090181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbc40fee70 0x7fdbc4194ca0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400b530 tx=0x7fdbb400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:13.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbc17fa700 1 -- 172.21.15.88:0/4057090181 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4011830 con 0x7fdbc40fee70 2023-07-14T17:43:13.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbc17fa700 1 -- 172.21.15.88:0/4057090181 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdbb4011e20 con 0x7fdbc40fee70 2023-07-14T17:43:13.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbca26f700 1 -- 172.21.15.88:0/4057090181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc41991b0 con 0x7fdbc40fee70 2023-07-14T17:43:13.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbc17fa700 1 -- 172.21.15.88:0/4057090181 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb400f2e0 con 0x7fdbc40fee70 2023-07-14T17:43:13.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.501+0000 7fdbca26f700 1 -- 172.21.15.88:0/4057090181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc41a1b00 con 0x7fdbc40fee70 2023-07-14T17:43:13.488 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.502+0000 7fdbca26f700 1 -- 172.21.15.88:0/4057090181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc404e340 con 0x7fdbc40fee70 2023-07-14T17:43:13.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.503+0000 7fdbc17fa700 1 -- 172.21.15.88:0/4057090181 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fdbb4011990 con 0x7fdbc40fee70 2023-07-14T17:43:13.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.504+0000 7fdbc17fa700 1 --2- 172.21.15.88:0/4057090181 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fdbac0a11e0 0x7fdbac0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:13.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.504+0000 7fdbc37fe700 1 --2- 172.21.15.88:0/4057090181 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fdbac0a11e0 0x7fdbac0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:13.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.504+0000 7fdbc17fa700 1 -- 172.21.15.88:0/4057090181 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fdbb40ccb10 con 0x7fdbc40fee70 2023-07-14T17:43:13.489 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.504+0000 7fdbc37fe700 1 --2- 172.21.15.88:0/4057090181 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fdbac0a11e0 0x7fdbac0a3690 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4196280 tx=0x7fdbb800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:13.490 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.507+0000 7fdbc17fa700 1 -- 172.21.15.88:0/4057090181 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdbb4096020 con 0x7fdbc40fee70 2023-07-14T17:43:13.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.691+0000 7fdbca26f700 1 -- 172.21.15.88:0/4057090181 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbc41a1d90 con 0x7fdbc40fee70 2023-07-14T17:43:13.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.693+0000 7fdbc17fa700 1 -- 172.21.15.88:0/4057090181 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23985 (secure 0 0 0) 0x7fdbb4099640 con 0x7fdbc40fee70 2023-07-14T17:43:13.675 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:43:13.675 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":86,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:42:46.934364+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:43:13.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.695+0000 7fdbca26f700 1 -- 172.21.15.88:0/4057090181 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fdbac0a11e0 msgr2=0x7fdbac0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:13.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.695+0000 7fdbca26f700 1 --2- 172.21.15.88:0/4057090181 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fdbac0a11e0 0x7fdbac0a3690 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdbc4196280 tx=0x7fdbb800a380 comp rx=0 tx=0).stop 2023-07-14T17:43:13.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.695+0000 7fdbca26f700 1 -- 172.21.15.88:0/4057090181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbc40fee70 msgr2=0x7fdbc4194ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:13.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.695+0000 7fdbca26f700 1 --2- 172.21.15.88:0/4057090181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbc40fee70 0x7fdbc4194ca0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400b530 tx=0x7fdbb400b840 comp rx=0 tx=0).stop 2023-07-14T17:43:13.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.695+0000 7fdbca26f700 1 -- 172.21.15.88:0/4057090181 shutdown_connections 2023-07-14T17:43:13.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.695+0000 7fdbca26f700 1 --2- 172.21.15.88:0/4057090181 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fdbc4103f10 0x7fdbc41985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:13.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.695+0000 7fdbca26f700 1 --2- 172.21.15.88:0/4057090181 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fdbac0a11e0 0x7fdbac0a3690 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:13.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.695+0000 7fdbca26f700 1 --2- 172.21.15.88:0/4057090181 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fdbc40fee70 0x7fdbc4194ca0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:13.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.695+0000 7fdbca26f700 1 --2- 172.21.15.88:0/4057090181 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fdbc40ff810 0x7fdbc41951e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:13.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.695+0000 7fdbca26f700 1 -- 172.21.15.88:0/4057090181 >> 172.21.15.88:0/4057090181 conn(0x7fdbc4075ae0 msgr2=0x7fdbc40fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:13.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.695+0000 7fdbca26f700 1 -- 172.21.15.88:0/4057090181 shutdown_connections 2023-07-14T17:43:13.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:13.695+0000 7fdbca26f700 1 -- 172.21.15.88:0/4057090181 wait complete. 2023-07-14T17:43:14.047 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:13 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/4057090181' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:43:14.130 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-07-14T17:43:14.131 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T17:43:14.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:13 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/4057090181' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:43:14.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:13 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/4057090181' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:43:14.678 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:15.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:14 smithi088 ceph-mon[127465]: pgmap v271: 129 pgs: 129 active+clean; 626 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.9 KiB/s wr, 22 op/s 2023-07-14T17:43:15.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:14 smithi107 ceph-mon[128999]: pgmap v271: 129 pgs: 129 active+clean; 626 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.9 KiB/s wr, 22 op/s 2023-07-14T17:43:15.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:14 smithi178 ceph-mon[129368]: pgmap v271: 129 pgs: 129 active+clean; 626 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 7.9 KiB/s wr, 22 op/s 2023-07-14T17:43:15.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.997+0000 7f820b45c700 1 -- 172.21.15.88:0/2523826711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8204104fd0 msgr2=0x7f820410cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:15.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.997+0000 7f820b45c700 1 --2- 172.21.15.88:0/2523826711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8204104fd0 0x7f820410cb50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f81fc00b0d0 tx=0x7f81fc00b3e0 comp rx=0 tx=0).stop 2023-07-14T17:43:15.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.997+0000 7f820b45c700 1 -- 172.21.15.88:0/2523826711 shutdown_connections 2023-07-14T17:43:15.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.997+0000 7f820b45c700 1 --2- 172.21.15.88:0/2523826711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8204104fd0 0x7f820410cb50 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:15.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.997+0000 7f820b45c700 1 --2- 172.21.15.88:0/2523826711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f82040fff30 0x7f8204100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:15.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.997+0000 7f820b45c700 1 --2- 172.21.15.88:0/2523826711 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f82041008d0 0x7f82041048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:15.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.997+0000 7f820b45c700 1 -- 172.21.15.88:0/2523826711 >> 172.21.15.88:0/2523826711 conn(0x7f82040fb790 msgr2=0x7f82040fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:15.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.997+0000 7f820b45c700 1 -- 172.21.15.88:0/2523826711 shutdown_connections 2023-07-14T17:43:15.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.997+0000 7f820b45c700 1 -- 172.21.15.88:0/2523826711 wait complete. 2023-07-14T17:43:15.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.998+0000 7f820b45c700 1 Processor -- start 2023-07-14T17:43:15.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.998+0000 7f820b45c700 1 -- start start 2023-07-14T17:43:15.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.998+0000 7f820b45c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f82040fff30 0x7f8204072940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:15.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.998+0000 7f820b45c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82041008d0 0x7f8204072e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:15.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.998+0000 7f820b45c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8204104fd0 0x7f8204073400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:15.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.998+0000 7f820b45c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82041a43a0 con 0x7f82041008d0 2023-07-14T17:43:15.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.998+0000 7f820b45c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82041a4510 con 0x7f8204104fd0 2023-07-14T17:43:15.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.998+0000 7f820b45c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82041a46d0 con 0x7f82040fff30 2023-07-14T17:43:15.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.998+0000 7f82089f7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82041008d0 0x7f8204072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:15.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.998+0000 7f82089f7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82041008d0 0x7f8204072e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42084/0 (socket says 172.21.15.88:42084) 2023-07-14T17:43:15.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.998+0000 7f82089f7700 1 -- 172.21.15.88:0/1708336517 learned_addr learned my addr 172.21.15.88:0/1708336517 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:15.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.998+0000 7f82099f9700 1 --2- 172.21.15.88:0/1708336517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8204104fd0 0x7f8204073400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:15.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.998+0000 7f82091f8700 1 --2- 172.21.15.88:0/1708336517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f82040fff30 0x7f8204072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:15.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.998+0000 7f82089f7700 1 -- 172.21.15.88:0/1708336517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f82040fff30 msgr2=0x7f8204072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:15.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.998+0000 7f82089f7700 1 --2- 172.21.15.88:0/1708336517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f82040fff30 0x7f8204072940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:15.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.998+0000 7f82089f7700 1 -- 172.21.15.88:0/1708336517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8204104fd0 msgr2=0x7f8204073400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:15.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.998+0000 7f82089f7700 1 --2- 172.21.15.88:0/1708336517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8204104fd0 0x7f8204073400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:15.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.998+0000 7f82089f7700 1 -- 172.21.15.88:0/1708336517 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81fc009d00 con 0x7f82041008d0 2023-07-14T17:43:15.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.999+0000 7f82089f7700 1 --2- 172.21.15.88:0/1708336517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82041008d0 0x7f8204072e80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f820000b500 tx=0x7f820000b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:15.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.999+0000 7f81fa7fc700 1 -- 172.21.15.88:0/1708336517 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8200011830 con 0x7f82041008d0 2023-07-14T17:43:15.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.999+0000 7f81fa7fc700 1 -- 172.21.15.88:0/1708336517 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8200011e20 con 0x7f82041008d0 2023-07-14T17:43:15.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.999+0000 7f820b45c700 1 -- 172.21.15.88:0/1708336517 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82041a49b0 con 0x7f82041008d0 2023-07-14T17:43:15.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.999+0000 7f81fa7fc700 1 -- 172.21.15.88:0/1708336517 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8200010db0 con 0x7f82041008d0 2023-07-14T17:43:15.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:15.999+0000 7f820b45c700 1 -- 172.21.15.88:0/1708336517 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82041a4eb0 con 0x7f82041008d0 2023-07-14T17:43:15.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.000+0000 7f820b45c700 1 -- 172.21.15.88:0/1708336517 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820404e340 con 0x7f82041008d0 2023-07-14T17:43:15.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.002+0000 7f81fa7fc700 1 -- 172.21.15.88:0/1708336517 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8200011990 con 0x7f82041008d0 2023-07-14T17:43:15.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.002+0000 7f81fa7fc700 1 --2- 172.21.15.88:0/1708336517 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f81f00a11e0 0x7f81f00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:15.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.002+0000 7f82091f8700 1 --2- 172.21.15.88:0/1708336517 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f81f00a11e0 0x7f81f00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:15.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.002+0000 7f81fa7fc700 1 -- 172.21.15.88:0/1708336517 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f82000ccab0 con 0x7f82041008d0 2023-07-14T17:43:15.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.003+0000 7f82091f8700 1 --2- 172.21.15.88:0/1708336517 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f81f00a11e0 0x7f81f00a3690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f81f4006fd0 tx=0x7f81f4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:15.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.005+0000 7f81fa7fc700 1 -- 172.21.15.88:0/1708336517 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8200095f60 con 0x7f82041008d0 2023-07-14T17:43:16.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.192+0000 7f820b45c700 1 -- 172.21.15.88:0/1708336517 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82041a51f0 con 0x7f82041008d0 2023-07-14T17:43:16.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.194+0000 7f81fa7fc700 1 -- 172.21.15.88:0/1708336517 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+23985 (secure 0 0 0) 0x7f8200099580 con 0x7f82041008d0 2023-07-14T17:43:16.176 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:43:16.176 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":86,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:42:46.934364+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:43:16.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.196+0000 7f820b45c700 1 -- 172.21.15.88:0/1708336517 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f81f00a11e0 msgr2=0x7f81f00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:16.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.196+0000 7f820b45c700 1 --2- 172.21.15.88:0/1708336517 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f81f00a11e0 0x7f81f00a3690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f81f4006fd0 tx=0x7f81f4008040 comp rx=0 tx=0).stop 2023-07-14T17:43:16.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.196+0000 7f820b45c700 1 -- 172.21.15.88:0/1708336517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82041008d0 msgr2=0x7f8204072e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:16.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.196+0000 7f820b45c700 1 --2- 172.21.15.88:0/1708336517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82041008d0 0x7f8204072e80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f820000b500 tx=0x7f820000b8c0 comp rx=0 tx=0).stop 2023-07-14T17:43:16.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.196+0000 7f820b45c700 1 -- 172.21.15.88:0/1708336517 shutdown_connections 2023-07-14T17:43:16.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.196+0000 7f820b45c700 1 --2- 172.21.15.88:0/1708336517 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8204104fd0 0x7f8204073400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:16.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.196+0000 7f820b45c700 1 --2- 172.21.15.88:0/1708336517 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f82040fff30 0x7f8204072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:16.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.196+0000 7f820b45c700 1 --2- 172.21.15.88:0/1708336517 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f81f00a11e0 0x7f81f00a3690 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:16.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.196+0000 7f820b45c700 1 --2- 172.21.15.88:0/1708336517 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82041008d0 0x7f8204072e80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:16.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.196+0000 7f820b45c700 1 -- 172.21.15.88:0/1708336517 >> 172.21.15.88:0/1708336517 conn(0x7f82040fb790 msgr2=0x7f82040fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:16.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.196+0000 7f820b45c700 1 -- 172.21.15.88:0/1708336517 shutdown_connections 2023-07-14T17:43:16.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:16.197+0000 7f820b45c700 1 -- 172.21.15.88:0/1708336517 wait complete. 2023-07-14T17:43:16.539 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2023-07-14T17:43:16.540 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2023-07-14T17:43:16.541 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2023-07-14T17:43:16.541 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2023-07-14T17:43:16.541 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2023-07-14T17:43:16.542 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2023-07-14T17:43:16.542 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2023-07-14T17:43:16.543 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2023-07-14T17:43:16.544 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2023-07-14T17:43:16.544 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph tell osd.9 flush_pg_stats 2023-07-14T17:43:16.545 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph tell osd.10 flush_pg_stats 2023-07-14T17:43:16.545 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph tell osd.11 flush_pg_stats 2023-07-14T17:43:16.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:16 smithi088 ceph-mon[127465]: pgmap v272: 129 pgs: 129 active+clean; 636 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.4 KiB/s wr, 21 op/s 2023-07-14T17:43:16.944 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:16 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1708336517' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:43:17.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:16 smithi178 ceph-mon[129368]: pgmap v272: 129 pgs: 129 active+clean; 636 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.4 KiB/s wr, 21 op/s 2023-07-14T17:43:17.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:16 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1708336517' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:43:17.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:16 smithi107 ceph-mon[128999]: pgmap v272: 129 pgs: 129 active+clean; 636 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.4 KiB/s wr, 21 op/s 2023-07-14T17:43:17.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:16 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1708336517' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:43:17.825 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:17.944 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:18.050 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:18.126 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:18.364 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:18.365 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:18.451 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:18.880 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:18.904 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:19.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:18 smithi088 ceph-mon[127465]: pgmap v273: 129 pgs: 129 active+clean; 636 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2023-07-14T17:43:19.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:18 smithi107 ceph-mon[128999]: pgmap v273: 129 pgs: 129 active+clean; 636 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2023-07-14T17:43:19.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:18 smithi178 ceph-mon[129368]: pgmap v273: 129 pgs: 129 active+clean; 636 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 2.2 KiB/s wr, 13 op/s 2023-07-14T17:43:19.670 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:19.869 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:19.947 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:20.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.778+0000 7fe53dbfe700 1 -- 172.21.15.88:0/91368485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe538101150 msgr2=0x7fe53810bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:20.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.778+0000 7fe53dbfe700 1 --2- 172.21.15.88:0/91368485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe538101150 0x7fe53810bb40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe530009fc0 tx=0x7fe53000b330 comp rx=0 tx=0).stop 2023-07-14T17:43:20.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.778+0000 7fe53dbfe700 1 -- 172.21.15.88:0/91368485 shutdown_connections 2023-07-14T17:43:20.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.778+0000 7fe53dbfe700 1 --2- 172.21.15.88:0/91368485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe538101150 0x7fe53810bb40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:20.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.778+0000 7fe53dbfe700 1 --2- 172.21.15.88:0/91368485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe5380fef40 0x7fe5380ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:20.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.778+0000 7fe53dbfe700 1 --2- 172.21.15.88:0/91368485 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5381007a0 0x7fe538100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.778+0000 7fe53dbfe700 1 -- 172.21.15.88:0/91368485 >> 172.21.15.88:0/91368485 conn(0x7fe538075c90 msgr2=0x7fe538076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.778+0000 7fe53dbfe700 1 -- 172.21.15.88:0/91368485 shutdown_connections 2023-07-14T17:43:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.778+0000 7fe53dbfe700 1 -- 172.21.15.88:0/91368485 wait complete. 2023-07-14T17:43:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.778+0000 7fe53dbfe700 1 Processor -- start 2023-07-14T17:43:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.779+0000 7fe53dbfe700 1 -- start start 2023-07-14T17:43:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.779+0000 7fe53dbfe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5380fef40 0x7fe538195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.779+0000 7fe53dbfe700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5381007a0 0x7fe538196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:20.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.779+0000 7fe53dbfe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe538101150 0x7fe53819a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:20.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.779+0000 7fe53dbfe700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53819ae00 con 0x7fe5380fef40 2023-07-14T17:43:20.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.779+0000 7fe53dbfe700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53819af70 con 0x7fe538101150 2023-07-14T17:43:20.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.779+0000 7fe53dbfe700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53819b130 con 0x7fe5381007a0 2023-07-14T17:43:20.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.779+0000 7fe5377fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5380fef40 0x7fe538195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:20.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.779+0000 7fe5377fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5380fef40 0x7fe538195e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42094/0 (socket says 172.21.15.88:42094) 2023-07-14T17:43:20.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.779+0000 7fe5377fe700 1 -- 172.21.15.88:0/889082154 learned_addr learned my addr 172.21.15.88:0/889082154 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:20.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.779+0000 7fe5377fe700 1 -- 172.21.15.88:0/889082154 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5381007a0 msgr2=0x7fe538196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:20.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.779+0000 7fe5377fe700 1 --2- 172.21.15.88:0/889082154 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe5381007a0 0x7fe538196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:20.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.779+0000 7fe5377fe700 1 -- 172.21.15.88:0/889082154 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe538101150 msgr2=0x7fe53819a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:20.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.779+0000 7fe5377fe700 1 --2- 172.21.15.88:0/889082154 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe538101150 0x7fe53819a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:20.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.779+0000 7fe5377fe700 1 -- 172.21.15.88:0/889082154 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5280095f0 con 0x7fe5380fef40 2023-07-14T17:43:20.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.779+0000 7fe5377fe700 1 --2- 172.21.15.88:0/889082154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5380fef40 0x7fe538195e40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe52800ec10 tx=0x7fe52800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:20.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.780+0000 7fe534ff9700 1 -- 172.21.15.88:0/889082154 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe528009a10 con 0x7fe5380fef40 2023-07-14T17:43:20.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.780+0000 7fe53dbfe700 1 -- 172.21.15.88:0/889082154 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe530009c40 con 0x7fe5380fef40 2023-07-14T17:43:20.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.780+0000 7fe534ff9700 1 -- 172.21.15.88:0/889082154 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe52800cde0 con 0x7fe5380fef40 2023-07-14T17:43:20.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.780+0000 7fe534ff9700 1 -- 172.21.15.88:0/889082154 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe528010430 con 0x7fe5380fef40 2023-07-14T17:43:20.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.780+0000 7fe53dbfe700 1 -- 172.21.15.88:0/889082154 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5381a2aa0 con 0x7fe5380fef40 2023-07-14T17:43:20.774 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.780+0000 7fe53dbfe700 1 -- 172.21.15.88:0/889082154 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe53810feb0 con 0x7fe5380fef40 2023-07-14T17:43:20.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.781+0000 7fe534ff9700 1 -- 172.21.15.88:0/889082154 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe528003fb0 con 0x7fe5380fef40 2023-07-14T17:43:20.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.782+0000 7fe534ff9700 1 --2- 172.21.15.88:0/889082154 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe5200a11e0 0x7fe5200a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:20.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.782+0000 7fe536ffd700 1 --2- 172.21.15.88:0/889082154 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe5200a11e0 0x7fe5200a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:20.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.782+0000 7fe534ff9700 1 -- 172.21.15.88:0/889082154 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe528014070 con 0x7fe5380fef40 2023-07-14T17:43:20.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.782+0000 7fe534ff9700 1 --2- 172.21.15.88:0/889082154 >> [v2:172.21.15.107:6808/2380989846,v1:172.21.15.107:6809/2380989846] conn(0x7fe5200abd50 0x7fe5200ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:20.775 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.782+0000 7fe534ff9700 1 -- 172.21.15.88:0/889082154 --> [v2:172.21.15.107:6808/2380989846,v1:172.21.15.107:6809/2380989846] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe5200ae850 con 0x7fe5200abd50 2023-07-14T17:43:20.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.782+0000 7fe534ff9700 1 -- 172.21.15.88:0/889082154 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5280cd730 con 0x7fe5380fef40 2023-07-14T17:43:20.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.782+0000 7fe536ffd700 1 --2- 172.21.15.88:0/889082154 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe5200a11e0 0x7fe5200a3690 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe52c005950 tx=0x7fe52c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:20.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.783+0000 7fe537fff700 1 --2- 172.21.15.88:0/889082154 >> [v2:172.21.15.107:6808/2380989846,v1:172.21.15.107:6809/2380989846] conn(0x7fe5200abd50 0x7fe5200ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:20.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.783+0000 7fe537fff700 1 --2- 172.21.15.88:0/889082154 >> [v2:172.21.15.107:6808/2380989846,v1:172.21.15.107:6809/2380989846] conn(0x7fe5200abd50 0x7fe5200ae180 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:20.776 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.784+0000 7fe534ff9700 1 -- 172.21.15.88:0/889082154 <== osd.4 v2:172.21.15.107:6808/2380989846 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fe5200ae850 con 0x7fe5200abd50 2023-07-14T17:43:20.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.800+0000 7fe53dbfe700 1 -- 172.21.15.88:0/889082154 --> [v2:172.21.15.107:6808/2380989846,v1:172.21.15.107:6809/2380989846] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe53804e340 con 0x7fe5200abd50 2023-07-14T17:43:20.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.800+0000 7fe534ff9700 1 -- 172.21.15.88:0/889082154 <== osd.4 v2:172.21.15.107:6808/2380989846 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe53804e340 con 0x7fe5200abd50 2023-07-14T17:43:20.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.801+0000 7fe53dbfe700 1 -- 172.21.15.88:0/889082154 >> [v2:172.21.15.107:6808/2380989846,v1:172.21.15.107:6809/2380989846] conn(0x7fe5200abd50 msgr2=0x7fe5200ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:20.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.801+0000 7fe53dbfe700 1 --2- 172.21.15.88:0/889082154 >> [v2:172.21.15.107:6808/2380989846,v1:172.21.15.107:6809/2380989846] conn(0x7fe5200abd50 0x7fe5200ae180 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:20.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.801+0000 7fe53dbfe700 1 -- 172.21.15.88:0/889082154 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe5200a11e0 msgr2=0x7fe5200a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:20.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.801+0000 7fe53dbfe700 1 --2- 172.21.15.88:0/889082154 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fe5200a11e0 0x7fe5200a3690 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe52c005950 tx=0x7fe52c0058e0 comp rx=0 tx=0).stop 2023-07-14T17:43:20.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.801+0000 7fe53dbfe700 1 -- 172.21.15.88:0/889082154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5380fef40 msgr2=0x7fe538195e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:20.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.801+0000 7fe53dbfe700 1 --2- 172.21.15.88:0/889082154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5380fef40 0x7fe538195e40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe52800ec10 tx=0x7fe52800c2d0 comp rx=0 tx=0).stop 2023-07-14T17:43:20.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.801+0000 7fe537fff700 1 -- 172.21.15.88:0/889082154 reap_dead start 2023-07-14T17:43:20.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.801+0000 7fe53dbfe700 1 -- 172.21.15.88:0/889082154 shutdown_connections 2023-07-14T17:43:20.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.801+0000 7fe53dbfe700 1 -- 172.21.15.88:0/889082154 >> 172.21.15.88:0/889082154 conn(0x7fe538075c90 msgr2=0x7fe5380fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:20.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.801+0000 7fe53dbfe700 1 -- 172.21.15.88:0/889082154 shutdown_connections 2023-07-14T17:43:20.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:20.801+0000 7fe53dbfe700 1 -- 172.21.15.88:0/889082154 wait complete. 2023-07-14T17:43:21.157 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:20 smithi088 ceph-mon[127465]: pgmap v274: 129 pgs: 129 active+clean; 636 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2023-07-14T17:43:21.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:20 smithi107 ceph-mon[128999]: pgmap v274: 129 pgs: 129 active+clean; 636 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2023-07-14T17:43:21.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:20 smithi178 ceph-mon[129368]: pgmap v274: 129 pgs: 129 active+clean; 636 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2023-07-14T17:43:22.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.155+0000 7ff1d2ced700 1 -- 172.21.15.88:0/989679101 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1cc1008d0 msgr2=0x7ff1cc1048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:22.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.155+0000 7ff1d2ced700 1 --2- 172.21.15.88:0/989679101 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1cc1008d0 0x7ff1cc1048a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff1b80099c0 tx=0x7ff1b8009cd0 comp rx=0 tx=0).stop 2023-07-14T17:43:22.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.155+0000 7ff1d2ced700 1 -- 172.21.15.88:0/989679101 shutdown_connections 2023-07-14T17:43:22.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.155+0000 7ff1d2ced700 1 --2- 172.21.15.88:0/989679101 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1cc104fd0 0x7ff1cc10cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:22.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.155+0000 7ff1d2ced700 1 --2- 172.21.15.88:0/989679101 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff1cc0fff30 0x7ff1cc100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:22.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.155+0000 7ff1d2ced700 1 --2- 172.21.15.88:0/989679101 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1cc1008d0 0x7ff1cc1048a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:22.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.155+0000 7ff1d2ced700 1 -- 172.21.15.88:0/989679101 >> 172.21.15.88:0/989679101 conn(0x7ff1cc0fb790 msgr2=0x7ff1cc0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:22.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.155+0000 7ff1d2ced700 1 -- 172.21.15.88:0/989679101 shutdown_connections 2023-07-14T17:43:22.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.155+0000 7ff1d2ced700 1 -- 172.21.15.88:0/989679101 wait complete. 2023-07-14T17:43:22.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.156+0000 7ff1d2ced700 1 Processor -- start 2023-07-14T17:43:22.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.156+0000 7ff1d2ced700 1 -- start start 2023-07-14T17:43:22.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.156+0000 7ff1d2ced700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1cc0fff30 0x7ff1cc195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:22.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.156+0000 7ff1d2ced700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1cc1008d0 0x7ff1cc1962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:22.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.156+0000 7ff1d2ced700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff1cc104fd0 0x7ff1cc19a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:22.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.156+0000 7ff1d2ced700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc19ad20 con 0x7ff1cc0fff30 2023-07-14T17:43:22.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.156+0000 7ff1d2ced700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc19ae90 con 0x7ff1cc1008d0 2023-07-14T17:43:22.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.156+0000 7ff1d2ced700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1cc19b050 con 0x7ff1cc104fd0 2023-07-14T17:43:22.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.156+0000 7ff1d0a89700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1cc0fff30 0x7ff1cc195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:22.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.156+0000 7ff1d0a89700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1cc0fff30 0x7ff1cc195d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42108/0 (socket says 172.21.15.88:42108) 2023-07-14T17:43:22.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.156+0000 7ff1d0a89700 1 -- 172.21.15.88:0/2582927241 learned_addr learned my addr 172.21.15.88:0/2582927241 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:22.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.157+0000 7ff1cbfff700 1 --2- 172.21.15.88:0/2582927241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1cc1008d0 0x7ff1cc1962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:22.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.157+0000 7ff1d0a89700 1 -- 172.21.15.88:0/2582927241 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff1cc104fd0 msgr2=0x7ff1cc19a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:22.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.157+0000 7ff1d0a89700 1 --2- 172.21.15.88:0/2582927241 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff1cc104fd0 0x7ff1cc19a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:22.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.157+0000 7ff1d0a89700 1 -- 172.21.15.88:0/2582927241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1cc1008d0 msgr2=0x7ff1cc1962a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:22.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.157+0000 7ff1d0a89700 1 --2- 172.21.15.88:0/2582927241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff1cc1008d0 0x7ff1cc1962a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:22.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.157+0000 7ff1d0a89700 1 -- 172.21.15.88:0/2582927241 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b80095f0 con 0x7ff1cc0fff30 2023-07-14T17:43:22.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.157+0000 7ff1d0a89700 1 --2- 172.21.15.88:0/2582927241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1cc0fff30 0x7ff1cc195d60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff1c000c920 tx=0x7ff1c000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:22.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.157+0000 7ff1c9ffb700 1 -- 172.21.15.88:0/2582927241 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c0007b00 con 0x7ff1cc0fff30 2023-07-14T17:43:22.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.157+0000 7ff1c9ffb700 1 -- 172.21.15.88:0/2582927241 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff1c0004cc0 con 0x7ff1cc0fff30 2023-07-14T17:43:22.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.157+0000 7ff1d2ced700 1 -- 172.21.15.88:0/2582927241 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1cc19b330 con 0x7ff1cc0fff30 2023-07-14T17:43:22.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.157+0000 7ff1c9ffb700 1 -- 172.21.15.88:0/2582927241 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1c0005590 con 0x7ff1cc0fff30 2023-07-14T17:43:22.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.157+0000 7ff1d2ced700 1 -- 172.21.15.88:0/2582927241 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1cc1a2b90 con 0x7ff1cc0fff30 2023-07-14T17:43:22.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.158+0000 7ff1d2ced700 1 -- 172.21.15.88:0/2582927241 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff1cc10fce0 con 0x7ff1cc0fff30 2023-07-14T17:43:22.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.159+0000 7ff1c9ffb700 1 -- 172.21.15.88:0/2582927241 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff1c0004750 con 0x7ff1cc0fff30 2023-07-14T17:43:22.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.159+0000 7ff1c9ffb700 1 --2- 172.21.15.88:0/2582927241 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff1bc0a11e0 0x7ff1bc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:22.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.160+0000 7ff1cbfff700 1 --2- 172.21.15.88:0/2582927241 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff1bc0a11e0 0x7ff1bc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:22.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.160+0000 7ff1c9ffb700 1 -- 172.21.15.88:0/2582927241 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff1c00ccde0 con 0x7ff1cc0fff30 2023-07-14T17:43:22.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.160+0000 7ff1c9ffb700 1 --2- 172.21.15.88:0/2582927241 >> [v2:172.21.15.107:6816/2214437741,v1:172.21.15.107:6817/2214437741] conn(0x7ff1bc0abd50 0x7ff1bc0ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:22.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.160+0000 7ff1c9ffb700 1 -- 172.21.15.88:0/2582927241 --> [v2:172.21.15.107:6816/2214437741,v1:172.21.15.107:6817/2214437741] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1bc0ae850 con 0x7ff1bc0abd50 2023-07-14T17:43:22.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.160+0000 7ff1c9ffb700 1 -- 172.21.15.88:0/2582927241 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1c00cd1f0 con 0x7ff1cc0fff30 2023-07-14T17:43:22.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.160+0000 7ff1cbfff700 1 --2- 172.21.15.88:0/2582927241 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff1bc0a11e0 0x7ff1bc0a3690 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8000c00 tx=0x7ff1b8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:22.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.160+0000 7ff1d128a700 1 --2- 172.21.15.88:0/2582927241 >> [v2:172.21.15.107:6816/2214437741,v1:172.21.15.107:6817/2214437741] conn(0x7ff1bc0abd50 0x7ff1bc0ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:22.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.161+0000 7ff1d128a700 1 --2- 172.21.15.88:0/2582927241 >> [v2:172.21.15.107:6816/2214437741,v1:172.21.15.107:6817/2214437741] conn(0x7ff1bc0abd50 0x7ff1bc0ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:22.505 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.161+0000 7ff1c9ffb700 1 -- 172.21.15.88:0/2582927241 <== osd.7 v2:172.21.15.107:6816/2214437741 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7ff1bc0ae850 con 0x7ff1bc0abd50 2023-07-14T17:43:22.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.178+0000 7ff1d2ced700 1 -- 172.21.15.88:0/2582927241 --> [v2:172.21.15.107:6816/2214437741,v1:172.21.15.107:6817/2214437741] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff1cc04e340 con 0x7ff1bc0abd50 2023-07-14T17:43:22.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.178+0000 7ff1c9ffb700 1 -- 172.21.15.88:0/2582927241 <== osd.7 v2:172.21.15.107:6816/2214437741 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff1cc04e340 con 0x7ff1bc0abd50 2023-07-14T17:43:22.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.179+0000 7ff1d2ced700 1 -- 172.21.15.88:0/2582927241 >> [v2:172.21.15.107:6816/2214437741,v1:172.21.15.107:6817/2214437741] conn(0x7ff1bc0abd50 msgr2=0x7ff1bc0ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:22.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.179+0000 7ff1d2ced700 1 --2- 172.21.15.88:0/2582927241 >> [v2:172.21.15.107:6816/2214437741,v1:172.21.15.107:6817/2214437741] conn(0x7ff1bc0abd50 0x7ff1bc0ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:22.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.179+0000 7ff1d2ced700 1 -- 172.21.15.88:0/2582927241 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff1bc0a11e0 msgr2=0x7ff1bc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:22.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.179+0000 7ff1d2ced700 1 --2- 172.21.15.88:0/2582927241 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff1bc0a11e0 0x7ff1bc0a3690 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8000c00 tx=0x7ff1b8005fb0 comp rx=0 tx=0).stop 2023-07-14T17:43:22.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.179+0000 7ff1d2ced700 1 -- 172.21.15.88:0/2582927241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1cc0fff30 msgr2=0x7ff1cc195d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:22.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.179+0000 7ff1d2ced700 1 --2- 172.21.15.88:0/2582927241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff1cc0fff30 0x7ff1cc195d60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff1c000c920 tx=0x7ff1c000cce0 comp rx=0 tx=0).stop 2023-07-14T17:43:22.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.179+0000 7ff1d128a700 1 -- 172.21.15.88:0/2582927241 reap_dead start 2023-07-14T17:43:22.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.179+0000 7ff1d2ced700 1 -- 172.21.15.88:0/2582927241 shutdown_connections 2023-07-14T17:43:22.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.179+0000 7ff1d2ced700 1 -- 172.21.15.88:0/2582927241 >> 172.21.15.88:0/2582927241 conn(0x7ff1cc0fb790 msgr2=0x7ff1cc0fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:22.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.179+0000 7ff1d2ced700 1 -- 172.21.15.88:0/2582927241 shutdown_connections 2023-07-14T17:43:22.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:22.179+0000 7ff1d2ced700 1 -- 172.21.15.88:0/2582927241 wait complete. 2023-07-14T17:43:23.043 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:22 smithi088 ceph-mon[127465]: pgmap v275: 129 pgs: 129 active+clean; 636 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2023-07-14T17:43:23.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:22 smithi107 ceph-mon[128999]: pgmap v275: 129 pgs: 129 active+clean; 636 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2023-07-14T17:43:23.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:22 smithi178 ceph-mon[129368]: pgmap v275: 129 pgs: 129 active+clean; 636 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2023-07-14T17:43:23.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.635+0000 7fb010e94700 1 -- 172.21.15.88:0/4271416056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb00c0ff7e0 msgr2=0x7fb00c1037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:23.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.635+0000 7fb010e94700 1 --2- 172.21.15.88:0/4271416056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb00c0ff7e0 0x7fb00c1037b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb0000099c0 tx=0x7fb000009cd0 comp rx=0 tx=0).stop 2023-07-14T17:43:23.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.635+0000 7fb010e94700 1 -- 172.21.15.88:0/4271416056 shutdown_connections 2023-07-14T17:43:23.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.635+0000 7fb010e94700 1 --2- 172.21.15.88:0/4271416056 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb00c103ee0 0x7fb00c10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:23.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.635+0000 7fb010e94700 1 --2- 172.21.15.88:0/4271416056 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb00c0fee40 0x7fb00c0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:23.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.635+0000 7fb010e94700 1 --2- 172.21.15.88:0/4271416056 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb00c0ff7e0 0x7fb00c1037b0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:23.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.635+0000 7fb010e94700 1 -- 172.21.15.88:0/4271416056 >> 172.21.15.88:0/4271416056 conn(0x7fb00c075ca0 msgr2=0x7fb00c0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:23.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.635+0000 7fb010e94700 1 -- 172.21.15.88:0/4271416056 shutdown_connections 2023-07-14T17:43:23.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.635+0000 7fb010e94700 1 -- 172.21.15.88:0/4271416056 wait complete. 2023-07-14T17:43:23.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.636+0000 7fb010e94700 1 Processor -- start 2023-07-14T17:43:23.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.636+0000 7fb010e94700 1 -- start start 2023-07-14T17:43:23.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.636+0000 7fb010e94700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb00c0fee40 0x7fb00c194900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:23.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.636+0000 7fb010e94700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb00c0ff7e0 0x7fb00c194e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:23.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.636+0000 7fb010e94700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb00c103ee0 0x7fb00c195380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:23.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.636+0000 7fb010e94700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c1a4140 con 0x7fb00c103ee0 2023-07-14T17:43:23.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.636+0000 7fb010e94700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c1a42b0 con 0x7fb00c0fee40 2023-07-14T17:43:23.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.636+0000 7fb010e94700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c1a4470 con 0x7fb00c0ff7e0 2023-07-14T17:43:23.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.637+0000 7fb00ad9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb00c103ee0 0x7fb00c195380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:23.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.637+0000 7fb00ad9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb00c103ee0 0x7fb00c195380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42124/0 (socket says 172.21.15.88:42124) 2023-07-14T17:43:23.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.637+0000 7fb00ad9d700 1 -- 172.21.15.88:0/2529897901 learned_addr learned my addr 172.21.15.88:0/2529897901 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:23.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.637+0000 7fb009d9b700 1 --2- 172.21.15.88:0/2529897901 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb00c0ff7e0 0x7fb00c194e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:23.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.637+0000 7fb00a59c700 1 --2- 172.21.15.88:0/2529897901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb00c0fee40 0x7fb00c194900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:23.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.637+0000 7fb00ad9d700 1 -- 172.21.15.88:0/2529897901 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb00c0ff7e0 msgr2=0x7fb00c194e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:23.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.637+0000 7fb00ad9d700 1 --2- 172.21.15.88:0/2529897901 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb00c0ff7e0 0x7fb00c194e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:23.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.637+0000 7fb00ad9d700 1 -- 172.21.15.88:0/2529897901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb00c0fee40 msgr2=0x7fb00c194900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:23.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.637+0000 7fb00ad9d700 1 --2- 172.21.15.88:0/2529897901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb00c0fee40 0x7fb00c194900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:23.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.637+0000 7fb00ad9d700 1 -- 172.21.15.88:0/2529897901 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0000095f0 con 0x7fb00c103ee0 2023-07-14T17:43:23.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.637+0000 7fb00a59c700 1 --2- 172.21.15.88:0/2529897901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb00c0fee40 0x7fb00c194900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:43:23.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.637+0000 7fb00ad9d700 1 --2- 172.21.15.88:0/2529897901 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb00c103ee0 0x7fb00c195380 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb004007f10 tx=0x7fb00400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:23.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.637+0000 7faffb7fe700 1 -- 172.21.15.88:0/2529897901 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb004017070 con 0x7fb00c103ee0 2023-07-14T17:43:23.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.637+0000 7fb010e94700 1 -- 172.21.15.88:0/2529897901 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00c1a4750 con 0x7fb00c103ee0 2023-07-14T17:43:23.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.637+0000 7faffb7fe700 1 -- 172.21.15.88:0/2529897901 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb00400f040 con 0x7fb00c103ee0 2023-07-14T17:43:23.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.637+0000 7faffb7fe700 1 -- 172.21.15.88:0/2529897901 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb004005710 con 0x7fb00c103ee0 2023-07-14T17:43:23.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.637+0000 7fb010e94700 1 -- 172.21.15.88:0/2529897901 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb00c1a4bd0 con 0x7fb00c103ee0 2023-07-14T17:43:23.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.638+0000 7fb010e94700 1 -- 172.21.15.88:0/2529897901 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb00c04e340 con 0x7fb00c103ee0 2023-07-14T17:43:23.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.639+0000 7faffb7fe700 1 -- 172.21.15.88:0/2529897901 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb004005870 con 0x7fb00c103ee0 2023-07-14T17:43:23.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.640+0000 7faffb7fe700 1 --2- 172.21.15.88:0/2529897901 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7faff40a11e0 0x7faff40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:23.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.640+0000 7fb00a59c700 1 --2- 172.21.15.88:0/2529897901 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7faff40a11e0 0x7faff40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:23.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.640+0000 7faffb7fe700 1 -- 172.21.15.88:0/2529897901 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb0040cca70 con 0x7fb00c103ee0 2023-07-14T17:43:23.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.640+0000 7faffb7fe700 1 --2- 172.21.15.88:0/2529897901 >> [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128] conn(0x7faff40abd50 0x7faff40ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:23.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.640+0000 7faffb7fe700 1 -- 172.21.15.88:0/2529897901 --> [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faff40ae850 con 0x7faff40abd50 2023-07-14T17:43:23.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.640+0000 7faffb7fe700 1 -- 172.21.15.88:0/2529897901 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0040ccde0 con 0x7fb00c103ee0 2023-07-14T17:43:23.626 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.640+0000 7fb009d9b700 1 --2- 172.21.15.88:0/2529897901 >> [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128] conn(0x7faff40abd50 0x7faff40ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:23.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.640+0000 7fb00a59c700 1 --2- 172.21.15.88:0/2529897901 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7faff40a11e0 0x7faff40a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faffc005950 tx=0x7faffc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:23.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.640+0000 7fb009d9b700 1 --2- 172.21.15.88:0/2529897901 >> [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128] conn(0x7faff40abd50 0x7faff40ae180 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:23.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.641+0000 7faffb7fe700 1 -- 172.21.15.88:0/2529897901 <== osd.6 v2:172.21.15.88:6818/1424514128 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7faff40ae850 con 0x7faff40abd50 2023-07-14T17:43:23.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.657+0000 7fb010e94700 1 -- 172.21.15.88:0/2529897901 --> [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb00c10ec30 con 0x7faff40abd50 2023-07-14T17:43:23.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.657+0000 7faffb7fe700 1 -- 172.21.15.88:0/2529897901 <== osd.6 v2:172.21.15.88:6818/1424514128 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb00001e040 con 0x7faff40abd50 2023-07-14T17:43:23.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.657+0000 7fb010e94700 1 -- 172.21.15.88:0/2529897901 >> [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128] conn(0x7faff40abd50 msgr2=0x7faff40ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:23.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.657+0000 7fb010e94700 1 --2- 172.21.15.88:0/2529897901 >> [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128] conn(0x7faff40abd50 0x7faff40ae180 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:23.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.657+0000 7fb010e94700 1 -- 172.21.15.88:0/2529897901 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7faff40a11e0 msgr2=0x7faff40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:23.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.657+0000 7fb010e94700 1 --2- 172.21.15.88:0/2529897901 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7faff40a11e0 0x7faff40a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faffc005950 tx=0x7faffc0058e0 comp rx=0 tx=0).stop 2023-07-14T17:43:23.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.657+0000 7fb010e94700 1 -- 172.21.15.88:0/2529897901 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb00c103ee0 msgr2=0x7fb00c195380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:23.641 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.657+0000 7fb010e94700 1 --2- 172.21.15.88:0/2529897901 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb00c103ee0 0x7fb00c195380 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb004007f10 tx=0x7fb00400d3b0 comp rx=0 tx=0).stop 2023-07-14T17:43:23.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.657+0000 7fb00ad9d700 1 -- 172.21.15.88:0/2529897901 reap_dead start 2023-07-14T17:43:23.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.657+0000 7fb010e94700 1 -- 172.21.15.88:0/2529897901 shutdown_connections 2023-07-14T17:43:23.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.657+0000 7fb010e94700 1 -- 172.21.15.88:0/2529897901 >> 172.21.15.88:0/2529897901 conn(0x7fb00c075ca0 msgr2=0x7fb00c0fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:23.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.658+0000 7fb010e94700 1 -- 172.21.15.88:0/2529897901 shutdown_connections 2023-07-14T17:43:23.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:23.658+0000 7fb010e94700 1 -- 172.21.15.88:0/2529897901 wait complete. 2023-07-14T17:43:24.803 INFO:teuthology.orchestra.run.smithi088.stdout:210453397540 2023-07-14T17:43:24.803 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.7 2023-07-14T17:43:25.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.806+0000 7f7e9b812700 1 -- 172.21.15.88:0/1108123112 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e940ffdb0 msgr2=0x7f7e9410ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:25.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.806+0000 7f7e9b812700 1 --2- 172.21.15.88:0/1108123112 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e940ffdb0 0x7f7e9410ba80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00b0d0 tx=0x7f7e8c00b3e0 comp rx=0 tx=0).stop 2023-07-14T17:43:25.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.806+0000 7f7e9b812700 1 -- 172.21.15.88:0/1108123112 shutdown_connections 2023-07-14T17:43:25.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.806+0000 7f7e9b812700 1 --2- 172.21.15.88:0/1108123112 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e940ffdb0 0x7f7e9410ba80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:25.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.806+0000 7f7e9b812700 1 --2- 172.21.15.88:0/1108123112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e94103be0 0x7f7e94103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.806+0000 7f7e9b812700 1 --2- 172.21.15.88:0/1108123112 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e940ff400 0x7f7e940ff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.806+0000 7f7e9b812700 1 -- 172.21.15.88:0/1108123112 >> 172.21.15.88:0/1108123112 conn(0x7f7e940751e0 msgr2=0x7f7e940755e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.806+0000 7f7e9b812700 1 -- 172.21.15.88:0/1108123112 shutdown_connections 2023-07-14T17:43:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.806+0000 7f7e9b812700 1 -- 172.21.15.88:0/1108123112 wait complete. 2023-07-14T17:43:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.807+0000 7f7e9b812700 1 Processor -- start 2023-07-14T17:43:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.807+0000 7f7e9b812700 1 -- start start 2023-07-14T17:43:25.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.807+0000 7f7e9b812700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e940ff400 0x7f7e9419bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.807+0000 7f7e9b812700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e940ffdb0 0x7f7e9419c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.807+0000 7f7e9b812700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e94103be0 0x7f7e94195b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.807+0000 7f7e9b812700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e941961b0 con 0x7f7e940ff400 2023-07-14T17:43:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.807+0000 7f7e9b812700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e94196320 con 0x7f7e94103be0 2023-07-14T17:43:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.807+0000 7f7e9b812700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e941964e0 con 0x7f7e940ffdb0 2023-07-14T17:43:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.807+0000 7f7e995ae700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e940ff400 0x7f7e9419bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.807+0000 7f7e995ae700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e940ff400 0x7f7e9419bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42136/0 (socket says 172.21.15.88:42136) 2023-07-14T17:43:25.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.807+0000 7f7e995ae700 1 -- 172.21.15.88:0/3785722363 learned_addr learned my addr 172.21.15.88:0/3785722363 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:25.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.807+0000 7f7e99daf700 1 --2- 172.21.15.88:0/3785722363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e94103be0 0x7f7e94195b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:25.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.808+0000 7f7e98dad700 1 --2- 172.21.15.88:0/3785722363 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e940ffdb0 0x7f7e9419c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:25.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.808+0000 7f7e995ae700 1 -- 172.21.15.88:0/3785722363 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e940ffdb0 msgr2=0x7f7e9419c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:25.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.808+0000 7f7e995ae700 1 --2- 172.21.15.88:0/3785722363 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7e940ffdb0 0x7f7e9419c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:25.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.808+0000 7f7e995ae700 1 -- 172.21.15.88:0/3785722363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e94103be0 msgr2=0x7f7e94195b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:25.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.808+0000 7f7e995ae700 1 --2- 172.21.15.88:0/3785722363 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7e94103be0 0x7f7e94195b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:25.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.808+0000 7f7e995ae700 1 -- 172.21.15.88:0/3785722363 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8c009d00 con 0x7f7e940ff400 2023-07-14T17:43:25.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.808+0000 7f7e995ae700 1 --2- 172.21.15.88:0/3785722363 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e940ff400 0x7f7e9419bac0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7e9000d950 tx=0x7f7e9000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:25.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.808+0000 7f7e8a7fc700 1 -- 172.21.15.88:0/3785722363 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e90009940 con 0x7f7e940ff400 2023-07-14T17:43:25.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.808+0000 7f7e8a7fc700 1 -- 172.21.15.88:0/3785722363 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7e90010460 con 0x7f7e940ff400 2023-07-14T17:43:25.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.808+0000 7f7e9b812700 1 -- 172.21.15.88:0/3785722363 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e941967c0 con 0x7f7e940ff400 2023-07-14T17:43:25.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.808+0000 7f7e8a7fc700 1 -- 172.21.15.88:0/3785722363 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9000f620 con 0x7f7e940ff400 2023-07-14T17:43:25.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.808+0000 7f7e9b812700 1 -- 172.21.15.88:0/3785722363 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e941a2ab0 con 0x7f7e940ff400 2023-07-14T17:43:25.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.809+0000 7f7e9b812700 1 -- 172.21.15.88:0/3785722363 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7e9404e340 con 0x7f7e940ff400 2023-07-14T17:43:25.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.810+0000 7f7e8a7fc700 1 -- 172.21.15.88:0/3785722363 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7e900105d0 con 0x7f7e940ff400 2023-07-14T17:43:25.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.811+0000 7f7e8a7fc700 1 --2- 172.21.15.88:0/3785722363 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7e800a11e0 0x7f7e800a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:25.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.811+0000 7f7e8a7fc700 1 -- 172.21.15.88:0/3785722363 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7e900cd0b0 con 0x7f7e940ff400 2023-07-14T17:43:25.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.811+0000 7f7e98dad700 1 --2- 172.21.15.88:0/3785722363 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7e800a11e0 0x7f7e800a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:25.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.811+0000 7f7e8a7fc700 1 --2- 172.21.15.88:0/3785722363 >> [v2:172.21.15.178:6800/4021325462,v1:172.21.15.178:6801/4021325462] conn(0x7f7e800abd50 0x7f7e800ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:25.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.811+0000 7f7e8a7fc700 1 -- 172.21.15.88:0/3785722363 --> [v2:172.21.15.178:6800/4021325462,v1:172.21.15.178:6801/4021325462] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e800ae850 con 0x7f7e800abd50 2023-07-14T17:43:25.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.811+0000 7f7e8a7fc700 1 -- 172.21.15.88:0/3785722363 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e900cd4c0 con 0x7f7e940ff400 2023-07-14T17:43:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.811+0000 7f7e98dad700 1 --2- 172.21.15.88:0/3785722363 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7e800a11e0 0x7f7e800a3690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7e84005ea0 tx=0x7f7e84005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.811+0000 7f7e99daf700 1 --2- 172.21.15.88:0/3785722363 >> [v2:172.21.15.178:6800/4021325462,v1:172.21.15.178:6801/4021325462] conn(0x7f7e800abd50 0x7f7e800ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.812+0000 7f7e99daf700 1 --2- 172.21.15.88:0/3785722363 >> [v2:172.21.15.178:6800/4021325462,v1:172.21.15.178:6801/4021325462] conn(0x7f7e800abd50 0x7f7e800ae180 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.813+0000 7f7e8a7fc700 1 -- 172.21.15.88:0/3785722363 <== osd.2 v2:172.21.15.178:6800/4021325462 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f7e800ae850 con 0x7f7e800abd50 2023-07-14T17:43:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.830+0000 7f7e9b812700 1 -- 172.21.15.88:0/3785722363 --> [v2:172.21.15.178:6800/4021325462,v1:172.21.15.178:6801/4021325462] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7e9410ed00 con 0x7f7e800abd50 2023-07-14T17:43:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.830+0000 7f7e8a7fc700 1 -- 172.21.15.88:0/3785722363 <== osd.2 v2:172.21.15.178:6800/4021325462 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7e8c009b80 con 0x7f7e800abd50 2023-07-14T17:43:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.831+0000 7f7e9b812700 1 -- 172.21.15.88:0/3785722363 >> [v2:172.21.15.178:6800/4021325462,v1:172.21.15.178:6801/4021325462] conn(0x7f7e800abd50 msgr2=0x7f7e800ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.831+0000 7f7e9b812700 1 --2- 172.21.15.88:0/3785722363 >> [v2:172.21.15.178:6800/4021325462,v1:172.21.15.178:6801/4021325462] conn(0x7f7e800abd50 0x7f7e800ae180 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.831+0000 7f7e9b812700 1 -- 172.21.15.88:0/3785722363 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7e800a11e0 msgr2=0x7f7e800a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.831+0000 7f7e9b812700 1 --2- 172.21.15.88:0/3785722363 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7e800a11e0 0x7f7e800a3690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7e84005ea0 tx=0x7f7e84005e30 comp rx=0 tx=0).stop 2023-07-14T17:43:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.831+0000 7f7e9b812700 1 -- 172.21.15.88:0/3785722363 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e940ff400 msgr2=0x7f7e9419bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.831+0000 7f7e9b812700 1 --2- 172.21.15.88:0/3785722363 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7e940ff400 0x7f7e9419bac0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7e9000d950 tx=0x7f7e9000dc60 comp rx=0 tx=0).stop 2023-07-14T17:43:25.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.831+0000 7f7e99daf700 1 -- 172.21.15.88:0/3785722363 reap_dead start 2023-07-14T17:43:25.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.831+0000 7f7e9b812700 1 -- 172.21.15.88:0/3785722363 shutdown_connections 2023-07-14T17:43:25.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.831+0000 7f7e9b812700 1 -- 172.21.15.88:0/3785722363 >> 172.21.15.88:0/3785722363 conn(0x7f7e940751e0 msgr2=0x7f7e94102820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:25.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.831+0000 7f7e9b812700 1 -- 172.21.15.88:0/3785722363 shutdown_connections 2023-07-14T17:43:25.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:24.831+0000 7f7e9b812700 1 -- 172.21.15.88:0/3785722363 wait complete. 2023-07-14T17:43:25.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:24 smithi088 ceph-mon[127465]: pgmap v276: 129 pgs: 129 active+clean; 636 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2023-07-14T17:43:25.215 INFO:teuthology.orchestra.run.smithi088.stdout:137438953520 2023-07-14T17:43:25.216 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.4 2023-07-14T17:43:25.217 INFO:teuthology.orchestra.run.smithi088.stdout:188978561064 2023-07-14T17:43:25.218 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.6 2023-07-14T17:43:25.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:24 smithi107 ceph-mon[128999]: pgmap v276: 129 pgs: 129 active+clean; 636 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2023-07-14T17:43:25.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:24 smithi178 ceph-mon[129368]: pgmap v276: 129 pgs: 129 active+clean; 636 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2023-07-14T17:43:25.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.589+0000 7f4813e27700 1 -- 172.21.15.88:0/770096429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c10a3a0 msgr2=0x7f480c10c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:25.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.589+0000 7f4813e27700 1 --2- 172.21.15.88:0/770096429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c10a3a0 0x7f480c10c7f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f480400b0a0 tx=0x7f480400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:43:25.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.589+0000 7f4813e27700 1 -- 172.21.15.88:0/770096429 shutdown_connections 2023-07-14T17:43:25.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.589+0000 7f4813e27700 1 --2- 172.21.15.88:0/770096429 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c10a3a0 0x7f480c10c7f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:25.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.589+0000 7f4813e27700 1 --2- 172.21.15.88:0/770096429 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f480c100f20 0x7f480c1012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:25.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.589+0000 7f4813e27700 1 --2- 172.21.15.88:0/770096429 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f480c1018c0 0x7f480c109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:25.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.589+0000 7f4813e27700 1 -- 172.21.15.88:0/770096429 >> 172.21.15.88:0/770096429 conn(0x7f480c075c90 msgr2=0x7f480c076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:25.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.589+0000 7f4813e27700 1 -- 172.21.15.88:0/770096429 shutdown_connections 2023-07-14T17:43:25.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.589+0000 7f4813e27700 1 -- 172.21.15.88:0/770096429 wait complete. 2023-07-14T17:43:25.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.590+0000 7f4813e27700 1 Processor -- start 2023-07-14T17:43:25.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.590+0000 7f4813e27700 1 -- start start 2023-07-14T17:43:25.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.590+0000 7f4813e27700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f480c100f20 0x7f480c195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:25.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.590+0000 7f4813e27700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c1018c0 0x7f480c1963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:25.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.590+0000 7f4813e27700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f480c10a3a0 0x7f480c19a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:25.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.590+0000 7f4813e27700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480c19ae40 con 0x7f480c1018c0 2023-07-14T17:43:25.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.590+0000 7f4813e27700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480c19afb0 con 0x7f480c100f20 2023-07-14T17:43:25.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.590+0000 7f4813e27700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480c19b170 con 0x7f480c10a3a0 2023-07-14T17:43:25.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.590+0000 7f48113c2700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c1018c0 0x7f480c1963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:25.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.590+0000 7f48113c2700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c1018c0 0x7f480c1963c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42150/0 (socket says 172.21.15.88:42150) 2023-07-14T17:43:25.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.590+0000 7f48113c2700 1 -- 172.21.15.88:0/46621224 learned_addr learned my addr 172.21.15.88:0/46621224 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:25.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.591+0000 7f48123c4700 1 --2- 172.21.15.88:0/46621224 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f480c10a3a0 0x7f480c19a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:25.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.590+0000 7f4811bc3700 1 --2- 172.21.15.88:0/46621224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f480c100f20 0x7f480c195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:25.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.591+0000 7f48113c2700 1 -- 172.21.15.88:0/46621224 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f480c10a3a0 msgr2=0x7f480c19a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:25.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.591+0000 7f48113c2700 1 --2- 172.21.15.88:0/46621224 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f480c10a3a0 0x7f480c19a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:25.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.591+0000 7f48113c2700 1 -- 172.21.15.88:0/46621224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f480c100f20 msgr2=0x7f480c195e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:25.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.591+0000 7f48113c2700 1 --2- 172.21.15.88:0/46621224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f480c100f20 0x7f480c195e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:25.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.591+0000 7f48113c2700 1 -- 172.21.15.88:0/46621224 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4804009d00 con 0x7f480c1018c0 2023-07-14T17:43:25.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.591+0000 7f4811bc3700 1 --2- 172.21.15.88:0/46621224 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f480c100f20 0x7f480c195e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:43:25.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.591+0000 7f48113c2700 1 --2- 172.21.15.88:0/46621224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c1018c0 0x7f480c1963c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00d980 tx=0x7f47fc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:25.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.591+0000 7f4802ffd700 1 -- 172.21.15.88:0/46621224 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc009930 con 0x7f480c1018c0 2023-07-14T17:43:25.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.591+0000 7f4813e27700 1 -- 172.21.15.88:0/46621224 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f480c19b450 con 0x7f480c1018c0 2023-07-14T17:43:25.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.591+0000 7f4802ffd700 1 -- 172.21.15.88:0/46621224 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f47fc010460 con 0x7f480c1018c0 2023-07-14T17:43:25.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.591+0000 7f4802ffd700 1 -- 172.21.15.88:0/46621224 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc00f620 con 0x7f480c1018c0 2023-07-14T17:43:25.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.591+0000 7f4813e27700 1 -- 172.21.15.88:0/46621224 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f480c1a2cb0 con 0x7f480c1018c0 2023-07-14T17:43:25.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.592+0000 7f4813e27700 1 -- 172.21.15.88:0/46621224 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f480c04e340 con 0x7f480c1018c0 2023-07-14T17:43:25.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.595+0000 7f4802ffd700 1 -- 172.21.15.88:0/46621224 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f47fc0105d0 con 0x7f480c1018c0 2023-07-14T17:43:25.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.595+0000 7f4802ffd700 1 --2- 172.21.15.88:0/46621224 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f47f80a11e0 0x7f47f80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:25.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.596+0000 7f4802ffd700 1 -- 172.21.15.88:0/46621224 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f47fc0ccab0 con 0x7f480c1018c0 2023-07-14T17:43:25.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.596+0000 7f4811bc3700 1 --2- 172.21.15.88:0/46621224 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f47f80a11e0 0x7f47f80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:25.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.596+0000 7f4802ffd700 1 --2- 172.21.15.88:0/46621224 >> [v2:172.21.15.178:6824/1862417235,v1:172.21.15.178:6825/1862417235] conn(0x7f47f80abcd0 0x7f47f80ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:25.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.596+0000 7f4802ffd700 1 -- 172.21.15.88:0/46621224 --> [v2:172.21.15.178:6824/1862417235,v1:172.21.15.178:6825/1862417235] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47f80ae7d0 con 0x7f47f80abcd0 2023-07-14T17:43:25.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.596+0000 7f4802ffd700 1 -- 172.21.15.88:0/46621224 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47fc0ccec0 con 0x7f480c1018c0 2023-07-14T17:43:25.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.596+0000 7f48123c4700 1 --2- 172.21.15.88:0/46621224 >> [v2:172.21.15.178:6824/1862417235,v1:172.21.15.178:6825/1862417235] conn(0x7f47f80abcd0 0x7f47f80ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:25.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.596+0000 7f4811bc3700 1 --2- 172.21.15.88:0/46621224 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f47f80a11e0 0x7f47f80a3690 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4808005950 tx=0x7f48080058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:25.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.596+0000 7f48123c4700 1 --2- 172.21.15.88:0/46621224 >> [v2:172.21.15.178:6824/1862417235,v1:172.21.15.178:6825/1862417235] conn(0x7f47f80abcd0 0x7f47f80ae100 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:25.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.597+0000 7f4802ffd700 1 -- 172.21.15.88:0/46621224 <== osd.11 v2:172.21.15.178:6824/1862417235 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f47f80ae7d0 con 0x7f47f80abcd0 2023-07-14T17:43:25.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.613+0000 7f4813e27700 1 -- 172.21.15.88:0/46621224 --> [v2:172.21.15.178:6824/1862417235,v1:172.21.15.178:6825/1862417235] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f480c10fe70 con 0x7f47f80abcd0 2023-07-14T17:43:25.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.614+0000 7f4802ffd700 1 -- 172.21.15.88:0/46621224 <== osd.11 v2:172.21.15.178:6824/1862417235 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4804009b80 con 0x7f47f80abcd0 2023-07-14T17:43:25.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.614+0000 7f4813e27700 1 -- 172.21.15.88:0/46621224 >> [v2:172.21.15.178:6824/1862417235,v1:172.21.15.178:6825/1862417235] conn(0x7f47f80abcd0 msgr2=0x7f47f80ae100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:25.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.614+0000 7f4813e27700 1 --2- 172.21.15.88:0/46621224 >> [v2:172.21.15.178:6824/1862417235,v1:172.21.15.178:6825/1862417235] conn(0x7f47f80abcd0 0x7f47f80ae100 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:25.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.614+0000 7f4813e27700 1 -- 172.21.15.88:0/46621224 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f47f80a11e0 msgr2=0x7f47f80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:25.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.614+0000 7f4813e27700 1 --2- 172.21.15.88:0/46621224 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f47f80a11e0 0x7f47f80a3690 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4808005950 tx=0x7f48080058e0 comp rx=0 tx=0).stop 2023-07-14T17:43:25.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.614+0000 7f4813e27700 1 -- 172.21.15.88:0/46621224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c1018c0 msgr2=0x7f480c1963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:25.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.614+0000 7f4813e27700 1 --2- 172.21.15.88:0/46621224 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f480c1018c0 0x7f480c1963c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f47fc00d980 tx=0x7f47fc00dc90 comp rx=0 tx=0).stop 2023-07-14T17:43:25.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.614+0000 7f48123c4700 1 -- 172.21.15.88:0/46621224 reap_dead start 2023-07-14T17:43:25.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.614+0000 7f4813e27700 1 -- 172.21.15.88:0/46621224 shutdown_connections 2023-07-14T17:43:25.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.614+0000 7f4813e27700 1 -- 172.21.15.88:0/46621224 >> 172.21.15.88:0/46621224 conn(0x7f480c075c90 msgr2=0x7f480c0fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:25.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.615+0000 7f4813e27700 1 -- 172.21.15.88:0/46621224 shutdown_connections 2023-07-14T17:43:25.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:25.615+0000 7f4813e27700 1 -- 172.21.15.88:0/46621224 wait complete. 2023-07-14T17:43:26.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.573+0000 7fbdb396a700 1 -- 172.21.15.88:0/1251465561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdac0fe5e0 msgr2=0x7fbdac0fe9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:26.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.573+0000 7fbdb396a700 1 --2- 172.21.15.88:0/1251465561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdac0fe5e0 0x7fbdac0fe9b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fbda800aad0 tx=0x7fbda800ade0 comp rx=0 tx=0).stop 2023-07-14T17:43:26.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.573+0000 7fbdb396a700 1 -- 172.21.15.88:0/1251465561 shutdown_connections 2023-07-14T17:43:26.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.573+0000 7fbdb396a700 1 --2- 172.21.15.88:0/1251465561 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdac103680 0x7fbdac10b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:26.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.573+0000 7fbdb396a700 1 --2- 172.21.15.88:0/1251465561 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdac0fe5e0 0x7fbdac0fe9b0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:26.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.573+0000 7fbdb396a700 1 --2- 172.21.15.88:0/1251465561 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbdac0fef80 0x7fbdac102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:26.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.573+0000 7fbdb396a700 1 -- 172.21.15.88:0/1251465561 >> 172.21.15.88:0/1251465561 conn(0x7fbdac075280 msgr2=0x7fbdac075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:26.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.573+0000 7fbdb396a700 1 -- 172.21.15.88:0/1251465561 shutdown_connections 2023-07-14T17:43:26.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.573+0000 7fbdb396a700 1 -- 172.21.15.88:0/1251465561 wait complete. 2023-07-14T17:43:26.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.574+0000 7fbdb396a700 1 Processor -- start 2023-07-14T17:43:26.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.574+0000 7fbdb396a700 1 -- start start 2023-07-14T17:43:26.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.574+0000 7fbdb396a700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbdac0fe5e0 0x7fbdac0ffff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:26.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.574+0000 7fbdb396a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdac0fef80 0x7fbdac100530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:26.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.574+0000 7fbdb396a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdac103680 0x7fbdac100ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:26.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.574+0000 7fbdb396a700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac1a4140 con 0x7fbdac0fef80 2023-07-14T17:43:26.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.574+0000 7fbdb396a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac1a42b0 con 0x7fbdac103680 2023-07-14T17:43:26.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.574+0000 7fbdb396a700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdac1a4470 con 0x7fbdac0fe5e0 2023-07-14T17:43:26.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.574+0000 7fbdb0f05700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdac0fef80 0x7fbdac100530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:26.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.574+0000 7fbdb0f05700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdac0fef80 0x7fbdac100530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42166/0 (socket says 172.21.15.88:42166) 2023-07-14T17:43:26.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.574+0000 7fbdb0f05700 1 -- 172.21.15.88:0/2509500105 learned_addr learned my addr 172.21.15.88:0/2509500105 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:26.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.574+0000 7fbdb1f07700 1 --2- 172.21.15.88:0/2509500105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdac103680 0x7fbdac100ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:26.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.575+0000 7fbdb1706700 1 --2- 172.21.15.88:0/2509500105 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbdac0fe5e0 0x7fbdac0ffff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:26.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.575+0000 7fbdb0f05700 1 -- 172.21.15.88:0/2509500105 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbdac0fe5e0 msgr2=0x7fbdac0ffff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:26.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.575+0000 7fbdb0f05700 1 --2- 172.21.15.88:0/2509500105 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbdac0fe5e0 0x7fbdac0ffff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:26.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.575+0000 7fbdb0f05700 1 -- 172.21.15.88:0/2509500105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdac103680 msgr2=0x7fbdac100ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:26.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.575+0000 7fbdb0f05700 1 --2- 172.21.15.88:0/2509500105 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fbdac103680 0x7fbdac100ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:26.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.575+0000 7fbdb0f05700 1 -- 172.21.15.88:0/2509500105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda800a700 con 0x7fbdac0fef80 2023-07-14T17:43:26.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.575+0000 7fbdb1706700 1 --2- 172.21.15.88:0/2509500105 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fbdac0fe5e0 0x7fbdac0ffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:43:26.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.575+0000 7fbdb0f05700 1 --2- 172.21.15.88:0/2509500105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdac0fef80 0x7fbdac100530 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00c8f0 tx=0x7fbd9c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:26.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.575+0000 7fbda27fc700 1 -- 172.21.15.88:0/2509500105 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c007940 con 0x7fbdac0fef80 2023-07-14T17:43:26.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.575+0000 7fbdb396a700 1 -- 172.21.15.88:0/2509500105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdac1a4750 con 0x7fbdac0fef80 2023-07-14T17:43:26.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.575+0000 7fbda27fc700 1 -- 172.21.15.88:0/2509500105 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fbd9c00f450 con 0x7fbdac0fef80 2023-07-14T17:43:26.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.575+0000 7fbda27fc700 1 -- 172.21.15.88:0/2509500105 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9c018660 con 0x7fbdac0fef80 2023-07-14T17:43:26.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.575+0000 7fbdb396a700 1 -- 172.21.15.88:0/2509500105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdac1a4c50 con 0x7fbdac0fef80 2023-07-14T17:43:26.562 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.576+0000 7fbdb396a700 1 -- 172.21.15.88:0/2509500105 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbdac10db60 con 0x7fbdac0fef80 2023-07-14T17:43:26.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.578+0000 7fbda27fc700 1 -- 172.21.15.88:0/2509500105 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fbd9c00f5c0 con 0x7fbdac0fef80 2023-07-14T17:43:26.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.578+0000 7fbda27fc700 1 --2- 172.21.15.88:0/2509500105 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbd980a11e0 0x7fbd980a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:26.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.578+0000 7fbdb1706700 1 --2- 172.21.15.88:0/2509500105 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbd980a11e0 0x7fbd980a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:26.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.578+0000 7fbda27fc700 1 -- 172.21.15.88:0/2509500105 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbd9c0cce70 con 0x7fbdac0fef80 2023-07-14T17:43:26.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.579+0000 7fbda27fc700 1 --2- 172.21.15.88:0/2509500105 >> [v2:172.21.15.88:6826/3211834989,v1:172.21.15.88:6827/3211834989] conn(0x7fbd980abd50 0x7fbd980ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:26.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.579+0000 7fbda27fc700 1 -- 172.21.15.88:0/2509500105 --> [v2:172.21.15.88:6826/3211834989,v1:172.21.15.88:6827/3211834989] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd980ae850 con 0x7fbd980abd50 2023-07-14T17:43:26.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.579+0000 7fbda27fc700 1 -- 172.21.15.88:0/2509500105 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd9c0cd1e0 con 0x7fbdac0fef80 2023-07-14T17:43:26.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.579+0000 7fbdb1f07700 1 --2- 172.21.15.88:0/2509500105 >> [v2:172.21.15.88:6826/3211834989,v1:172.21.15.88:6827/3211834989] conn(0x7fbd980abd50 0x7fbd980ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:26.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.579+0000 7fbdb1706700 1 --2- 172.21.15.88:0/2509500105 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbd980a11e0 0x7fbd980a3690 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbda800a6d0 tx=0x7fbda800a620 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:26.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.579+0000 7fbdb1f07700 1 --2- 172.21.15.88:0/2509500105 >> [v2:172.21.15.88:6826/3211834989,v1:172.21.15.88:6827/3211834989] conn(0x7fbd980abd50 0x7fbd980ae180 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:26.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.580+0000 7fbda27fc700 1 -- 172.21.15.88:0/2509500105 <== osd.9 v2:172.21.15.88:6826/3211834989 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fbd980ae850 con 0x7fbd980abd50 2023-07-14T17:43:26.579 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.596+0000 7fbdb396a700 1 -- 172.21.15.88:0/2509500105 --> [v2:172.21.15.88:6826/3211834989,v1:172.21.15.88:6827/3211834989] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbdac04e340 con 0x7fbd980abd50 2023-07-14T17:43:26.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.597+0000 7fbda27fc700 1 -- 172.21.15.88:0/2509500105 <== osd.9 v2:172.21.15.88:6826/3211834989 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fbdac04e340 con 0x7fbd980abd50 2023-07-14T17:43:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.597+0000 7fbdb396a700 1 -- 172.21.15.88:0/2509500105 >> [v2:172.21.15.88:6826/3211834989,v1:172.21.15.88:6827/3211834989] conn(0x7fbd980abd50 msgr2=0x7fbd980ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.597+0000 7fbdb396a700 1 --2- 172.21.15.88:0/2509500105 >> [v2:172.21.15.88:6826/3211834989,v1:172.21.15.88:6827/3211834989] conn(0x7fbd980abd50 0x7fbd980ae180 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.597+0000 7fbdb396a700 1 -- 172.21.15.88:0/2509500105 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbd980a11e0 msgr2=0x7fbd980a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.597+0000 7fbdb396a700 1 --2- 172.21.15.88:0/2509500105 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fbd980a11e0 0x7fbd980a3690 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbda800a6d0 tx=0x7fbda800a620 comp rx=0 tx=0).stop 2023-07-14T17:43:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.597+0000 7fbdb396a700 1 -- 172.21.15.88:0/2509500105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdac0fef80 msgr2=0x7fbdac100530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.597+0000 7fbdb396a700 1 --2- 172.21.15.88:0/2509500105 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbdac0fef80 0x7fbdac100530 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c00c8f0 tx=0x7fbd9c00cc00 comp rx=0 tx=0).stop 2023-07-14T17:43:26.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.597+0000 7fbdb1f07700 1 -- 172.21.15.88:0/2509500105 reap_dead start 2023-07-14T17:43:26.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.597+0000 7fbdb396a700 1 -- 172.21.15.88:0/2509500105 shutdown_connections 2023-07-14T17:43:26.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.597+0000 7fbdb396a700 1 -- 172.21.15.88:0/2509500105 >> 172.21.15.88:0/2509500105 conn(0x7fbdac075280 msgr2=0x7fbdac102370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:26.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.597+0000 7fbdb396a700 1 -- 172.21.15.88:0/2509500105 shutdown_connections 2023-07-14T17:43:26.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.597+0000 7fbdb396a700 1 -- 172.21.15.88:0/2509500105 wait complete. 2023-07-14T17:43:26.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.743+0000 7f4cf3524700 1 -- 172.21.15.88:0/1119342649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cec10a270 msgr2=0x7f4cec10c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:26.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.743+0000 7f4cf3524700 1 --2- 172.21.15.88:0/1119342649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cec10a270 0x7f4cec10c6b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f4ce400b0a0 tx=0x7f4ce400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:43:26.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.743+0000 7f4cf3524700 1 -- 172.21.15.88:0/1119342649 shutdown_connections 2023-07-14T17:43:26.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.743+0000 7f4cf3524700 1 --2- 172.21.15.88:0/1119342649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cec10a270 0x7f4cec10c6b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:26.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.743+0000 7f4cf3524700 1 --2- 172.21.15.88:0/1119342649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec100f20 0x7f4cec1012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:26.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.743+0000 7f4cf3524700 1 --2- 172.21.15.88:0/1119342649 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4cec101830 0x7f4cec109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:26.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.743+0000 7f4cf3524700 1 -- 172.21.15.88:0/1119342649 >> 172.21.15.88:0/1119342649 conn(0x7f4cec075ca0 msgr2=0x7f4cec0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:26.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.743+0000 7f4cf3524700 1 -- 172.21.15.88:0/1119342649 shutdown_connections 2023-07-14T17:43:26.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.743+0000 7f4cf3524700 1 -- 172.21.15.88:0/1119342649 wait complete. 2023-07-14T17:43:26.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf3524700 1 Processor -- start 2023-07-14T17:43:26.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf3524700 1 -- start start 2023-07-14T17:43:26.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf3524700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4cec100f20 0x7f4cec195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:26.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf3524700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec101830 0x7f4cec196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:26.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf3524700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cec10a270 0x7f4cec19a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:26.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf3524700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec19adc0 con 0x7f4cec10a270 2023-07-14T17:43:26.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf3524700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec19af30 con 0x7f4cec101830 2023-07-14T17:43:26.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf3524700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec19b0f0 con 0x7f4cec100f20 2023-07-14T17:43:26.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf1ac1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cec10a270 0x7f4cec19a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:26.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf1ac1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cec10a270 0x7f4cec19a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42200/0 (socket says 172.21.15.88:42200) 2023-07-14T17:43:26.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf1ac1700 1 -- 172.21.15.88:0/3295269118 learned_addr learned my addr 172.21.15.88:0/3295269118 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:26.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf0abf700 1 --2- 172.21.15.88:0/3295269118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec101830 0x7f4cec196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:26.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf12c0700 1 --2- 172.21.15.88:0/3295269118 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4cec100f20 0x7f4cec195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:26.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf1ac1700 1 -- 172.21.15.88:0/3295269118 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4cec100f20 msgr2=0x7f4cec195e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:26.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf1ac1700 1 --2- 172.21.15.88:0/3295269118 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4cec100f20 0x7f4cec195e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:26.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf1ac1700 1 -- 172.21.15.88:0/3295269118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec101830 msgr2=0x7f4cec196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:26.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf1ac1700 1 --2- 172.21.15.88:0/3295269118 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4cec101830 0x7f4cec196340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:26.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf1ac1700 1 -- 172.21.15.88:0/3295269118 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce4009d00 con 0x7f4cec10a270 2023-07-14T17:43:26.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.744+0000 7f4cf1ac1700 1 --2- 172.21.15.88:0/3295269118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cec10a270 0x7f4cec19a7b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4ce400b830 tx=0x7f4ce40094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:26.731 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.745+0000 7f4ce27fc700 1 -- 172.21.15.88:0/3295269118 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce400e040 con 0x7f4cec10a270 2023-07-14T17:43:26.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.745+0000 7f4cf3524700 1 -- 172.21.15.88:0/3295269118 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cec19b370 con 0x7f4cec10a270 2023-07-14T17:43:26.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.745+0000 7f4ce27fc700 1 -- 172.21.15.88:0/3295269118 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4ce4003e90 con 0x7f4cec10a270 2023-07-14T17:43:26.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.745+0000 7f4ce27fc700 1 -- 172.21.15.88:0/3295269118 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce4004840 con 0x7f4cec10a270 2023-07-14T17:43:26.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.745+0000 7f4cf3524700 1 -- 172.21.15.88:0/3295269118 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cec1a2c30 con 0x7f4cec10a270 2023-07-14T17:43:26.732 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.746+0000 7f4cf3524700 1 -- 172.21.15.88:0/3295269118 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4cec10fcd0 con 0x7f4cec10a270 2023-07-14T17:43:26.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.747+0000 7f4ce27fc700 1 -- 172.21.15.88:0/3295269118 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4ce4003a00 con 0x7f4cec10a270 2023-07-14T17:43:26.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.747+0000 7f4ce27fc700 1 --2- 172.21.15.88:0/3295269118 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4cd80a11e0 0x7f4cd80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:26.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.747+0000 7f4cf12c0700 1 --2- 172.21.15.88:0/3295269118 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4cd80a11e0 0x7f4cd80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:26.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.747+0000 7f4ce27fc700 1 -- 172.21.15.88:0/3295269118 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4ce40cedf0 con 0x7f4cec10a270 2023-07-14T17:43:26.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.748+0000 7f4ce27fc700 1 --2- 172.21.15.88:0/3295269118 >> [v2:172.21.15.178:6808/4006439725,v1:172.21.15.178:6809/4006439725] conn(0x7f4cd80abd50 0x7f4cd80ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:26.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.748+0000 7f4ce27fc700 1 -- 172.21.15.88:0/3295269118 --> [v2:172.21.15.178:6808/4006439725,v1:172.21.15.178:6809/4006439725] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4cd80ae850 con 0x7f4cd80abd50 2023-07-14T17:43:26.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.748+0000 7f4ce27fc700 1 -- 172.21.15.88:0/3295269118 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ce40cf200 con 0x7f4cec10a270 2023-07-14T17:43:26.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.748+0000 7f4cf12c0700 1 --2- 172.21.15.88:0/3295269118 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4cd80a11e0 0x7f4cd80a3690 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8005950 tx=0x7f4ce800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:26.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.748+0000 7f4cf0abf700 1 --2- 172.21.15.88:0/3295269118 >> [v2:172.21.15.178:6808/4006439725,v1:172.21.15.178:6809/4006439725] conn(0x7f4cd80abd50 0x7f4cd80ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:26.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.748+0000 7f4cf0abf700 1 --2- 172.21.15.88:0/3295269118 >> [v2:172.21.15.178:6808/4006439725,v1:172.21.15.178:6809/4006439725] conn(0x7f4cd80abd50 0x7f4cd80ae180 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:26.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.749+0000 7f4ce27fc700 1 -- 172.21.15.88:0/3295269118 <== osd.5 v2:172.21.15.178:6808/4006439725 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f4cd80ae850 con 0x7f4cd80abd50 2023-07-14T17:43:26.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.769+0000 7f4cf3524700 1 -- 172.21.15.88:0/3295269118 --> [v2:172.21.15.178:6808/4006439725,v1:172.21.15.178:6809/4006439725] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4cec04e340 con 0x7f4cd80abd50 2023-07-14T17:43:26.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.770+0000 7f4ce27fc700 1 -- 172.21.15.88:0/3295269118 <== osd.5 v2:172.21.15.178:6808/4006439725 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4cec04e340 con 0x7f4cd80abd50 2023-07-14T17:43:26.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.770+0000 7f4cf3524700 1 -- 172.21.15.88:0/3295269118 >> [v2:172.21.15.178:6808/4006439725,v1:172.21.15.178:6809/4006439725] conn(0x7f4cd80abd50 msgr2=0x7f4cd80ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:26.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.770+0000 7f4cf3524700 1 --2- 172.21.15.88:0/3295269118 >> [v2:172.21.15.178:6808/4006439725,v1:172.21.15.178:6809/4006439725] conn(0x7f4cd80abd50 0x7f4cd80ae180 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:26.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.770+0000 7f4cf3524700 1 -- 172.21.15.88:0/3295269118 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4cd80a11e0 msgr2=0x7f4cd80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:26.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.770+0000 7f4cf3524700 1 --2- 172.21.15.88:0/3295269118 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4cd80a11e0 0x7f4cd80a3690 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8005950 tx=0x7f4ce800b410 comp rx=0 tx=0).stop 2023-07-14T17:43:26.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.771+0000 7f4cf3524700 1 -- 172.21.15.88:0/3295269118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cec10a270 msgr2=0x7f4cec19a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:26.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.771+0000 7f4cf3524700 1 --2- 172.21.15.88:0/3295269118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cec10a270 0x7f4cec19a7b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f4ce400b830 tx=0x7f4ce40094c0 comp rx=0 tx=0).stop 2023-07-14T17:43:26.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.771+0000 7f4cf1ac1700 1 -- 172.21.15.88:0/3295269118 reap_dead start 2023-07-14T17:43:26.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.771+0000 7f4cf3524700 1 -- 172.21.15.88:0/3295269118 shutdown_connections 2023-07-14T17:43:26.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.771+0000 7f4cf3524700 1 -- 172.21.15.88:0/3295269118 >> 172.21.15.88:0/3295269118 conn(0x7f4cec075ca0 msgr2=0x7f4cec10ac70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:26.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.771+0000 7f4cf3524700 1 -- 172.21.15.88:0/3295269118 shutdown_connections 2023-07-14T17:43:26.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.771+0000 7f4cf3524700 1 -- 172.21.15.88:0/3295269118 wait complete. 2023-07-14T17:43:26.941 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:26 smithi088 ceph-mon[127465]: pgmap v277: 129 pgs: 129 active+clean; 639 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2023-07-14T17:43:26.974 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:26 smithi107 ceph-mon[128999]: pgmap v277: 129 pgs: 129 active+clean; 639 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2023-07-14T17:43:27.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:26 smithi178 ceph-mon[129368]: pgmap v277: 129 pgs: 129 active+clean; 639 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2023-07-14T17:43:27.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.640+0000 7fb032626700 1 -- 172.21.15.88:0/2703139305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb02c0feab0 msgr2=0x7fb02c0fee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:27.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.640+0000 7fb032626700 1 --2- 172.21.15.88:0/2703139305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb02c0feab0 0x7fb02c0fee80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb01c009990 tx=0x7fb01c009ca0 comp rx=0 tx=0).stop 2023-07-14T17:43:27.256 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.640+0000 7fb032626700 1 -- 172.21.15.88:0/2703139305 shutdown_connections 2023-07-14T17:43:27.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.640+0000 7fb032626700 1 --2- 172.21.15.88:0/2703139305 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb02c10c660 0x7fb02c10ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:27.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.640+0000 7fb032626700 1 --2- 172.21.15.88:0/2703139305 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb02c0feab0 0x7fb02c0fee80 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:27.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.640+0000 7fb032626700 1 --2- 172.21.15.88:0/2703139305 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb02c0ff3c0 0x7fb02c10c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:27.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.640+0000 7fb032626700 1 -- 172.21.15.88:0/2703139305 >> 172.21.15.88:0/2703139305 conn(0x7fb02c075ca0 msgr2=0x7fb02c0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:27.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.640+0000 7fb032626700 1 -- 172.21.15.88:0/2703139305 shutdown_connections 2023-07-14T17:43:27.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.640+0000 7fb032626700 1 -- 172.21.15.88:0/2703139305 wait complete. 2023-07-14T17:43:27.257 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.640+0000 7fb032626700 1 Processor -- start 2023-07-14T17:43:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.640+0000 7fb032626700 1 -- start start 2023-07-14T17:43:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.641+0000 7fb032626700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb02c0feab0 0x7fb02c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.641+0000 7fb032626700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb02c0ff3c0 0x7fb02c198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.641+0000 7fb032626700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb02c10c660 0x7fb02c19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.641+0000 7fb032626700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c19cf00 con 0x7fb02c0ff3c0 2023-07-14T17:43:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.641+0000 7fb032626700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c19d070 con 0x7fb02c10c660 2023-07-14T17:43:27.258 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.641+0000 7fb032626700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c19d230 con 0x7fb02c0feab0 2023-07-14T17:43:27.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.641+0000 7fb02b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb02c0ff3c0 0x7fb02c198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:27.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.641+0000 7fb02b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb02c0ff3c0 0x7fb02c198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42182/0 (socket says 172.21.15.88:42182) 2023-07-14T17:43:27.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.641+0000 7fb02b7fe700 1 -- 172.21.15.88:0/2487867022 learned_addr learned my addr 172.21.15.88:0/2487867022 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:27.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.641+0000 7fb02bfff700 1 --2- 172.21.15.88:0/2487867022 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb02c0feab0 0x7fb02c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:27.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.641+0000 7fb030bc3700 1 --2- 172.21.15.88:0/2487867022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb02c10c660 0x7fb02c19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:27.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.641+0000 7fb02b7fe700 1 -- 172.21.15.88:0/2487867022 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb02c0feab0 msgr2=0x7fb02c197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:27.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.641+0000 7fb02b7fe700 1 --2- 172.21.15.88:0/2487867022 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb02c0feab0 0x7fb02c197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:27.259 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.641+0000 7fb02b7fe700 1 -- 172.21.15.88:0/2487867022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb02c10c660 msgr2=0x7fb02c19c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:27.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.641+0000 7fb02b7fe700 1 --2- 172.21.15.88:0/2487867022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb02c10c660 0x7fb02c19c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:27.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.641+0000 7fb02b7fe700 1 -- 172.21.15.88:0/2487867022 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c0095f0 con 0x7fb02c0ff3c0 2023-07-14T17:43:27.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.641+0000 7fb030bc3700 1 --2- 172.21.15.88:0/2487867022 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb02c10c660 0x7fb02c19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:43:27.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.642+0000 7fb02b7fe700 1 --2- 172.21.15.88:0/2487867022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb02c0ff3c0 0x7fb02c198480 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb02000e9a0 tx=0x7fb02000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:27.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.642+0000 7fb0297fa700 1 -- 172.21.15.88:0/2487867022 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb020009790 con 0x7fb02c0ff3c0 2023-07-14T17:43:27.260 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.642+0000 7fb032626700 1 -- 172.21.15.88:0/2487867022 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb02c19d510 con 0x7fb02c0ff3c0 2023-07-14T17:43:27.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.642+0000 7fb0297fa700 1 -- 172.21.15.88:0/2487867022 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb02000cbd0 con 0x7fb02c0ff3c0 2023-07-14T17:43:27.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.642+0000 7fb0297fa700 1 -- 172.21.15.88:0/2487867022 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb020018680 con 0x7fb02c0ff3c0 2023-07-14T17:43:27.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.642+0000 7fb032626700 1 -- 172.21.15.88:0/2487867022 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb02c1a4d70 con 0x7fb02c0ff3c0 2023-07-14T17:43:27.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.643+0000 7fb032626700 1 -- 172.21.15.88:0/2487867022 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb02c04e340 con 0x7fb02c0ff3c0 2023-07-14T17:43:27.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.644+0000 7fb0297fa700 1 -- 172.21.15.88:0/2487867022 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb02000cd40 con 0x7fb02c0ff3c0 2023-07-14T17:43:27.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.644+0000 7fb0297fa700 1 --2- 172.21.15.88:0/2487867022 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb0140a11e0 0x7fb0140a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:27.261 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.644+0000 7fb0297fa700 1 -- 172.21.15.88:0/2487867022 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb020014070 con 0x7fb02c0ff3c0 2023-07-14T17:43:27.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.644+0000 7fb02bfff700 1 --2- 172.21.15.88:0/2487867022 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb0140a11e0 0x7fb0140a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:27.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.645+0000 7fb0297fa700 1 --2- 172.21.15.88:0/2487867022 >> [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618] conn(0x7fb0140abd50 0x7fb0140ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:27.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.645+0000 7fb0297fa700 1 -- 172.21.15.88:0/2487867022 --> [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb0140ae850 con 0x7fb0140abd50 2023-07-14T17:43:27.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.645+0000 7fb0297fa700 1 -- 172.21.15.88:0/2487867022 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0200cd640 con 0x7fb02c0ff3c0 2023-07-14T17:43:27.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.645+0000 7fb030bc3700 1 --2- 172.21.15.88:0/2487867022 >> [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618] conn(0x7fb0140abd50 0x7fb0140ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:27.262 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.645+0000 7fb02bfff700 1 --2- 172.21.15.88:0/2487867022 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb0140a11e0 0x7fb0140a3690 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb01c006010 tx=0x7fb01c00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:27.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.645+0000 7fb030bc3700 1 --2- 172.21.15.88:0/2487867022 >> [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618] conn(0x7fb0140abd50 0x7fb0140ae180 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:27.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.646+0000 7fb0297fa700 1 -- 172.21.15.88:0/2487867022 <== osd.3 v2:172.21.15.88:6810/3836966618 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb0140ae850 con 0x7fb0140abd50 2023-07-14T17:43:27.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.676+0000 7fb032626700 1 -- 172.21.15.88:0/2487867022 --> [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb02c111fb0 con 0x7fb0140abd50 2023-07-14T17:43:27.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.677+0000 7fb0297fa700 1 -- 172.21.15.88:0/2487867022 <== osd.3 v2:172.21.15.88:6810/3836966618 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb02400f040 con 0x7fb0140abd50 2023-07-14T17:43:27.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.677+0000 7fb032626700 1 -- 172.21.15.88:0/2487867022 >> [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618] conn(0x7fb0140abd50 msgr2=0x7fb0140ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:27.263 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.677+0000 7fb032626700 1 --2- 172.21.15.88:0/2487867022 >> [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618] conn(0x7fb0140abd50 0x7fb0140ae180 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:27.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.677+0000 7fb032626700 1 -- 172.21.15.88:0/2487867022 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb0140a11e0 msgr2=0x7fb0140a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:27.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.677+0000 7fb032626700 1 --2- 172.21.15.88:0/2487867022 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb0140a11e0 0x7fb0140a3690 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb01c006010 tx=0x7fb01c00b260 comp rx=0 tx=0).stop 2023-07-14T17:43:27.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.677+0000 7fb032626700 1 -- 172.21.15.88:0/2487867022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb02c0ff3c0 msgr2=0x7fb02c198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:27.264 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.677+0000 7fb032626700 1 --2- 172.21.15.88:0/2487867022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb02c0ff3c0 0x7fb02c198480 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb02000e9a0 tx=0x7fb02000ed60 comp rx=0 tx=0).stop 2023-07-14T17:43:27.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.677+0000 7fb030bc3700 1 -- 172.21.15.88:0/2487867022 reap_dead start 2023-07-14T17:43:27.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.677+0000 7fb032626700 1 -- 172.21.15.88:0/2487867022 shutdown_connections 2023-07-14T17:43:27.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.677+0000 7fb032626700 1 -- 172.21.15.88:0/2487867022 >> 172.21.15.88:0/2487867022 conn(0x7fb02c075ca0 msgr2=0x7fb02c0fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:27.265 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.677+0000 7fb032626700 1 -- 172.21.15.88:0/2487867022 shutdown_connections 2023-07-14T17:43:27.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:26.677+0000 7fb032626700 1 -- 172.21.15.88:0/2487867022 wait complete. 2023-07-14T17:43:27.486 INFO:teuthology.orchestra.run.smithi088.stdout:158913789998 2023-07-14T17:43:27.486 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.5 2023-07-14T17:43:27.566 INFO:teuthology.orchestra.run.smithi088.stdout:77309411385 2023-07-14T17:43:27.567 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.2 2023-07-14T17:43:27.915 INFO:teuthology.orchestra.run.smithi088.stdout:274877906972 2023-07-14T17:43:27.915 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.9 2023-07-14T17:43:28.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.054+0000 7fca0908e700 1 -- 172.21.15.88:0/3669818723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca0410c660 msgr2=0x7fca0410ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.054+0000 7fca0908e700 1 --2- 172.21.15.88:0/3669818723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca0410c660 0x7fca0410ea40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc00b0a0 tx=0x7fc9fc00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.055+0000 7fca0908e700 1 -- 172.21.15.88:0/3669818723 shutdown_connections 2023-07-14T17:43:28.038 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.055+0000 7fca0908e700 1 --2- 172.21.15.88:0/3669818723 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca0410c660 0x7fca0410ea40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.055+0000 7fca0908e700 1 --2- 172.21.15.88:0/3669818723 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca040feab0 0x7fca040fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.055+0000 7fca0908e700 1 --2- 172.21.15.88:0/3669818723 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca040ff3c0 0x7fca0410c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.055+0000 7fca0908e700 1 -- 172.21.15.88:0/3669818723 >> 172.21.15.88:0/3669818723 conn(0x7fca04075ca0 msgr2=0x7fca040760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:28.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.055+0000 7fca0908e700 1 -- 172.21.15.88:0/3669818723 shutdown_connections 2023-07-14T17:43:28.039 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.055+0000 7fca0908e700 1 -- 172.21.15.88:0/3669818723 wait complete. 2023-07-14T17:43:28.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.056+0000 7fca0908e700 1 Processor -- start 2023-07-14T17:43:28.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.056+0000 7fca0908e700 1 -- start start 2023-07-14T17:43:28.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.056+0000 7fca0908e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca040feab0 0x7fca04197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.056+0000 7fca0908e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca040ff3c0 0x7fca04198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.056+0000 7fca0908e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca0410c660 0x7fca0419c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.056+0000 7fca0908e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0419cfb0 con 0x7fca0410c660 2023-07-14T17:43:28.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.056+0000 7fca0908e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0419d120 con 0x7fca040feab0 2023-07-14T17:43:28.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.056+0000 7fca0908e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0419d2e0 con 0x7fca040ff3c0 2023-07-14T17:43:28.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.056+0000 7fca0359e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca0410c660 0x7fca0419c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.056+0000 7fca0359e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca0410c660 0x7fca0419c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42230/0 (socket says 172.21.15.88:42230) 2023-07-14T17:43:28.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.056+0000 7fca0359e700 1 -- 172.21.15.88:0/2696596424 learned_addr learned my addr 172.21.15.88:0/2696596424 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:28.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.056+0000 7fca02d9d700 1 --2- 172.21.15.88:0/2696596424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca040feab0 0x7fca04197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.056+0000 7fca0259c700 1 --2- 172.21.15.88:0/2696596424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca040ff3c0 0x7fca04198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.057+0000 7fca0359e700 1 -- 172.21.15.88:0/2696596424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca040ff3c0 msgr2=0x7fca04198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.057+0000 7fca0359e700 1 --2- 172.21.15.88:0/2696596424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca040ff3c0 0x7fca04198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.057+0000 7fca0359e700 1 -- 172.21.15.88:0/2696596424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca040feab0 msgr2=0x7fca04197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.057+0000 7fca0359e700 1 --2- 172.21.15.88:0/2696596424 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fca040feab0 0x7fca04197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.057+0000 7fca0359e700 1 -- 172.21.15.88:0/2696596424 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9fc009d00 con 0x7fca0410c660 2023-07-14T17:43:28.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.057+0000 7fca0259c700 1 --2- 172.21.15.88:0/2696596424 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fca040ff3c0 0x7fca04198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:43:28.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.057+0000 7fca0359e700 1 --2- 172.21.15.88:0/2696596424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca0410c660 0x7fca0419c9a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc009610 tx=0x7fc9fc008e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:28.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.057+0000 7fc9ebfff700 1 -- 172.21.15.88:0/2696596424 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc00e040 con 0x7fca0410c660 2023-07-14T17:43:28.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.057+0000 7fca0908e700 1 -- 172.21.15.88:0/2696596424 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0419d560 con 0x7fca0410c660 2023-07-14T17:43:28.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.057+0000 7fca0908e700 1 -- 172.21.15.88:0/2696596424 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca041a4e20 con 0x7fca0410c660 2023-07-14T17:43:28.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.057+0000 7fc9ebfff700 1 -- 172.21.15.88:0/2696596424 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc9fc01d470 con 0x7fca0410c660 2023-07-14T17:43:28.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.057+0000 7fc9ebfff700 1 -- 172.21.15.88:0/2696596424 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc0052a0 con 0x7fca0410c660 2023-07-14T17:43:28.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.058+0000 7fca0908e700 1 -- 172.21.15.88:0/2696596424 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc9f0000ff0 con 0x7fca0410c660 2023-07-14T17:43:28.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.059+0000 7fc9ebfff700 1 -- 172.21.15.88:0/2696596424 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc9fc01d5e0 con 0x7fca0410c660 2023-07-14T17:43:28.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.060+0000 7fc9ebfff700 1 --2- 172.21.15.88:0/2696596424 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc9ec0a1260 0x7fc9ec0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.060+0000 7fca02d9d700 1 --2- 172.21.15.88:0/2696596424 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc9ec0a1260 0x7fc9ec0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.060+0000 7fc9ebfff700 1 -- 172.21.15.88:0/2696596424 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc9fc0cde70 con 0x7fca0410c660 2023-07-14T17:43:28.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.060+0000 7fc9ebfff700 1 --2- 172.21.15.88:0/2696596424 >> [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341] conn(0x7fc9ec0abdd0 0x7fc9ec0ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.060+0000 7fc9ebfff700 1 -- 172.21.15.88:0/2696596424 --> [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9ec0ae8d0 con 0x7fc9ec0abdd0 2023-07-14T17:43:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.060+0000 7fc9ebfff700 1 -- 172.21.15.88:0/2696596424 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9fc0d3500 con 0x7fca0410c660 2023-07-14T17:43:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.060+0000 7fca02d9d700 1 --2- 172.21.15.88:0/2696596424 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc9ec0a1260 0x7fc9ec0a3710 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4009c60 tx=0x7fc9f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.060+0000 7fca0259c700 1 --2- 172.21.15.88:0/2696596424 >> [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341] conn(0x7fc9ec0abdd0 0x7fc9ec0ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.061+0000 7fca0259c700 1 --2- 172.21.15.88:0/2696596424 >> [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341] conn(0x7fc9ec0abdd0 0x7fc9ec0ae200 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:28.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.062+0000 7fc9ebfff700 1 -- 172.21.15.88:0/2696596424 <== osd.0 v2:172.21.15.88:6802/826983341 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc9ec0ae8d0 con 0x7fc9ec0abdd0 2023-07-14T17:43:28.062 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.080+0000 7fca0908e700 1 -- 172.21.15.88:0/2696596424 --> [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc9f0002d70 con 0x7fc9ec0abdd0 2023-07-14T17:43:28.063 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.080+0000 7fc9ebfff700 1 -- 172.21.15.88:0/2696596424 <== osd.0 v2:172.21.15.88:6802/826983341 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc9f0002d70 con 0x7fc9ec0abdd0 2023-07-14T17:43:28.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.081+0000 7fca0908e700 1 -- 172.21.15.88:0/2696596424 >> [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341] conn(0x7fc9ec0abdd0 msgr2=0x7fc9ec0ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.064 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.081+0000 7fca0908e700 1 --2- 172.21.15.88:0/2696596424 >> [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341] conn(0x7fc9ec0abdd0 0x7fc9ec0ae200 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.081+0000 7fca0908e700 1 -- 172.21.15.88:0/2696596424 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc9ec0a1260 msgr2=0x7fc9ec0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.081+0000 7fca0908e700 1 --2- 172.21.15.88:0/2696596424 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc9ec0a1260 0x7fc9ec0a3710 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4009c60 tx=0x7fc9f4009260 comp rx=0 tx=0).stop 2023-07-14T17:43:28.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.081+0000 7fca0908e700 1 -- 172.21.15.88:0/2696596424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca0410c660 msgr2=0x7fca0419c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.081+0000 7fca0908e700 1 --2- 172.21.15.88:0/2696596424 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca0410c660 0x7fca0419c9a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc009610 tx=0x7fc9fc008e50 comp rx=0 tx=0).stop 2023-07-14T17:43:28.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.081+0000 7fca0359e700 1 -- 172.21.15.88:0/2696596424 reap_dead start 2023-07-14T17:43:28.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.081+0000 7fca0908e700 1 -- 172.21.15.88:0/2696596424 shutdown_connections 2023-07-14T17:43:28.065 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.081+0000 7fca0908e700 1 -- 172.21.15.88:0/2696596424 >> 172.21.15.88:0/2696596424 conn(0x7fca04075ca0 msgr2=0x7fca040fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:28.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.081+0000 7fca0908e700 1 -- 172.21.15.88:0/2696596424 shutdown_connections 2023-07-14T17:43:28.066 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.081+0000 7fca0908e700 1 -- 172.21.15.88:0/2696596424 wait complete. 2023-07-14T17:43:28.090 INFO:teuthology.orchestra.run.smithi088.stdout:330712481812 2023-07-14T17:43:28.091 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.11 2023-07-14T17:43:28.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.198+0000 7f6e95232700 1 -- 172.21.15.88:0/382506808 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e90100f20 msgr2=0x7f6e901012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.198+0000 7f6e95232700 1 --2- 172.21.15.88:0/382506808 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e90100f20 0x7f6e901012f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6e80009990 tx=0x7f6e80009ca0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.198+0000 7f6e95232700 1 -- 172.21.15.88:0/382506808 shutdown_connections 2023-07-14T17:43:28.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.198+0000 7f6e95232700 1 --2- 172.21.15.88:0/382506808 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e9010a3a0 0x7f6e9010c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.198+0000 7f6e95232700 1 --2- 172.21.15.88:0/382506808 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e90100f20 0x7f6e901012f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.198+0000 7f6e95232700 1 --2- 172.21.15.88:0/382506808 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e901018c0 0x7f6e90109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.198+0000 7f6e95232700 1 -- 172.21.15.88:0/382506808 >> 172.21.15.88:0/382506808 conn(0x7f6e90075c90 msgr2=0x7f6e90076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:28.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.198+0000 7f6e95232700 1 -- 172.21.15.88:0/382506808 shutdown_connections 2023-07-14T17:43:28.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.198+0000 7f6e95232700 1 -- 172.21.15.88:0/382506808 wait complete. 2023-07-14T17:43:28.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.199+0000 7f6e95232700 1 Processor -- start 2023-07-14T17:43:28.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.199+0000 7f6e95232700 1 -- start start 2023-07-14T17:43:28.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.200+0000 7f6e95232700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e90100f20 0x7f6e90195d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.200+0000 7f6e95232700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e901018c0 0x7f6e90196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.200+0000 7f6e95232700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e9010a3a0 0x7f6e9019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.200+0000 7f6e95232700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9019acf0 con 0x7f6e9010a3a0 2023-07-14T17:43:28.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.200+0000 7f6e95232700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9019ae60 con 0x7f6e90100f20 2023-07-14T17:43:28.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.200+0000 7f6e95232700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9019b020 con 0x7f6e901018c0 2023-07-14T17:43:28.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.200+0000 7f6e8f59e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e9010a3a0 0x7f6e9019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.200+0000 7f6e8f59e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e9010a3a0 0x7f6e9019a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42242/0 (socket says 172.21.15.88:42242) 2023-07-14T17:43:28.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.200+0000 7f6e8f59e700 1 -- 172.21.15.88:0/166998046 learned_addr learned my addr 172.21.15.88:0/166998046 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:28.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.200+0000 7f6e8e59c700 1 --2- 172.21.15.88:0/166998046 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e901018c0 0x7f6e90196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.200+0000 7f6e8ed9d700 1 --2- 172.21.15.88:0/166998046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e90100f20 0x7f6e90195d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.200+0000 7f6e8f59e700 1 -- 172.21.15.88:0/166998046 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e901018c0 msgr2=0x7f6e90196270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.200+0000 7f6e8f59e700 1 --2- 172.21.15.88:0/166998046 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6e901018c0 0x7f6e90196270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.200+0000 7f6e8f59e700 1 -- 172.21.15.88:0/166998046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e90100f20 msgr2=0x7f6e90195d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.200+0000 7f6e8f59e700 1 --2- 172.21.15.88:0/166998046 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6e90100f20 0x7f6e90195d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.200+0000 7f6e8f59e700 1 -- 172.21.15.88:0/166998046 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e800095f0 con 0x7f6e9010a3a0 2023-07-14T17:43:28.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.200+0000 7f6e8f59e700 1 --2- 172.21.15.88:0/166998046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e9010a3a0 0x7f6e9019a6e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800ada0 tx=0x7f6e8800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:28.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.201+0000 7f6e77fff700 1 -- 172.21.15.88:0/166998046 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8800f070 con 0x7f6e9010a3a0 2023-07-14T17:43:28.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.201+0000 7f6e95232700 1 -- 172.21.15.88:0/166998046 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e9019b300 con 0x7f6e9010a3a0 2023-07-14T17:43:28.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.201+0000 7f6e77fff700 1 -- 172.21.15.88:0/166998046 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6e88007b30 con 0x7f6e9010a3a0 2023-07-14T17:43:28.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.201+0000 7f6e77fff700 1 -- 172.21.15.88:0/166998046 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e88005280 con 0x7f6e9010a3a0 2023-07-14T17:43:28.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.201+0000 7f6e95232700 1 -- 172.21.15.88:0/166998046 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e900fec80 con 0x7f6e9010a3a0 2023-07-14T17:43:28.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.202+0000 7f6e95232700 1 -- 172.21.15.88:0/166998046 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6e9004e340 con 0x7f6e9010a3a0 2023-07-14T17:43:28.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.203+0000 7f6e77fff700 1 -- 172.21.15.88:0/166998046 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6e8800c980 con 0x7f6e9010a3a0 2023-07-14T17:43:28.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.204+0000 7f6e77fff700 1 --2- 172.21.15.88:0/166998046 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6e780a11e0 0x7f6e780a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.204+0000 7f6e77fff700 1 -- 172.21.15.88:0/166998046 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6e880cccc0 con 0x7f6e9010a3a0 2023-07-14T17:43:28.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.204+0000 7f6e8ed9d700 1 --2- 172.21.15.88:0/166998046 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6e780a11e0 0x7f6e780a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.204+0000 7f6e77fff700 1 --2- 172.21.15.88:0/166998046 >> [v2:172.21.15.107:6800/1795455333,v1:172.21.15.107:6801/1795455333] conn(0x7f6e780abd50 0x7f6e780ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.204+0000 7f6e77fff700 1 -- 172.21.15.88:0/166998046 --> [v2:172.21.15.107:6800/1795455333,v1:172.21.15.107:6801/1795455333] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e780ae850 con 0x7f6e780abd50 2023-07-14T17:43:28.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.204+0000 7f6e77fff700 1 -- 172.21.15.88:0/166998046 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e880cd0d0 con 0x7f6e9010a3a0 2023-07-14T17:43:28.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.204+0000 7f6e8ed9d700 1 --2- 172.21.15.88:0/166998046 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6e780a11e0 0x7f6e780a3690 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6e8000b5c0 tx=0x7f6e80005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:28.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.204+0000 7f6e8e59c700 1 --2- 172.21.15.88:0/166998046 >> [v2:172.21.15.107:6800/1795455333,v1:172.21.15.107:6801/1795455333] conn(0x7f6e780abd50 0x7f6e780ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.205+0000 7f6e8e59c700 1 --2- 172.21.15.88:0/166998046 >> [v2:172.21.15.107:6800/1795455333,v1:172.21.15.107:6801/1795455333] conn(0x7f6e780abd50 0x7f6e780ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:28.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.206+0000 7f6e77fff700 1 -- 172.21.15.88:0/166998046 <== osd.1 v2:172.21.15.107:6800/1795455333 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f6e780ae850 con 0x7f6e780abd50 2023-07-14T17:43:28.205 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.222+0000 7f6e95232700 1 -- 172.21.15.88:0/166998046 --> [v2:172.21.15.107:6800/1795455333,v1:172.21.15.107:6801/1795455333] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6e9010feb0 con 0x7f6e780abd50 2023-07-14T17:43:28.206 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.223+0000 7f6e77fff700 1 -- 172.21.15.88:0/166998046 <== osd.1 v2:172.21.15.107:6800/1795455333 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6e8400a8a0 con 0x7f6e780abd50 2023-07-14T17:43:28.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.224+0000 7f6e95232700 1 -- 172.21.15.88:0/166998046 >> [v2:172.21.15.107:6800/1795455333,v1:172.21.15.107:6801/1795455333] conn(0x7f6e780abd50 msgr2=0x7f6e780ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.224+0000 7f6e95232700 1 --2- 172.21.15.88:0/166998046 >> [v2:172.21.15.107:6800/1795455333,v1:172.21.15.107:6801/1795455333] conn(0x7f6e780abd50 0x7f6e780ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.224+0000 7f6e95232700 1 -- 172.21.15.88:0/166998046 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6e780a11e0 msgr2=0x7f6e780a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.224+0000 7f6e95232700 1 --2- 172.21.15.88:0/166998046 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6e780a11e0 0x7f6e780a3690 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6e8000b5c0 tx=0x7f6e80005fb0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.224+0000 7f6e95232700 1 -- 172.21.15.88:0/166998046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e9010a3a0 msgr2=0x7f6e9019a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.224+0000 7f6e95232700 1 --2- 172.21.15.88:0/166998046 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6e9010a3a0 0x7f6e9019a6e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800ada0 tx=0x7f6e8800c330 comp rx=0 tx=0).stop 2023-07-14T17:43:28.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.225+0000 7f6e8f59e700 1 -- 172.21.15.88:0/166998046 reap_dead start 2023-07-14T17:43:28.209 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.225+0000 7f6e95232700 1 -- 172.21.15.88:0/166998046 shutdown_connections 2023-07-14T17:43:28.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.225+0000 7f6e95232700 1 -- 172.21.15.88:0/166998046 >> 172.21.15.88:0/166998046 conn(0x7f6e90075c90 msgr2=0x7f6e90101cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:28.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.225+0000 7f6e95232700 1 -- 172.21.15.88:0/166998046 shutdown_connections 2023-07-14T17:43:28.213 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.225+0000 7f6e95232700 1 -- 172.21.15.88:0/166998046 wait complete. 2023-07-14T17:43:28.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.916+0000 7f643a5d1700 1 -- 172.21.15.88:0/331594236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6434104c70 msgr2=0x7f6434105040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.916+0000 7f643a5d1700 1 --2- 172.21.15.88:0/331594236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6434104c70 0x7f6434105040 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6424009990 tx=0x7f6424009ca0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.916+0000 7f643a5d1700 1 -- 172.21.15.88:0/331594236 shutdown_connections 2023-07-14T17:43:28.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.916+0000 7f643a5d1700 1 --2- 172.21.15.88:0/331594236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64340febc0 0x7f643410cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.916+0000 7f643a5d1700 1 --2- 172.21.15.88:0/331594236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6434104c70 0x7f6434105040 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.916+0000 7f643a5d1700 1 --2- 172.21.15.88:0/331594236 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f64340fe210 0x7f64340fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.916+0000 7f643a5d1700 1 -- 172.21.15.88:0/331594236 >> 172.21.15.88:0/331594236 conn(0x7f6434075280 msgr2=0x7f6434075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:28.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.916+0000 7f643a5d1700 1 -- 172.21.15.88:0/331594236 shutdown_connections 2023-07-14T17:43:28.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.916+0000 7f643a5d1700 1 -- 172.21.15.88:0/331594236 wait complete. 2023-07-14T17:43:28.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.917+0000 7f643a5d1700 1 Processor -- start 2023-07-14T17:43:28.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.917+0000 7f643a5d1700 1 -- start start 2023-07-14T17:43:28.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f643a5d1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64340fe210 0x7f64340ff3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f643a5d1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64340febc0 0x7f64340ff930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f643a5d1700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6434104c70 0x7f64340ffeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.558 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f643a5d1700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64341a40b0 con 0x7f64340febc0 2023-07-14T17:43:28.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f643a5d1700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64341a4220 con 0x7f64340fe210 2023-07-14T17:43:28.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f643a5d1700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64341a43e0 con 0x7f6434104c70 2023-07-14T17:43:28.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f64337fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64340febc0 0x7f64340ff930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f64337fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64340febc0 0x7f64340ff930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42212/0 (socket says 172.21.15.88:42212) 2023-07-14T17:43:28.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f64337fe700 1 -- 172.21.15.88:0/2045094591 learned_addr learned my addr 172.21.15.88:0/2045094591 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:28.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f6438b6e700 1 --2- 172.21.15.88:0/2045094591 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6434104c70 0x7f64340ffeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.559 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f6433fff700 1 --2- 172.21.15.88:0/2045094591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64340fe210 0x7f64340ff3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f64337fe700 1 -- 172.21.15.88:0/2045094591 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6434104c70 msgr2=0x7f64340ffeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f64337fe700 1 --2- 172.21.15.88:0/2045094591 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6434104c70 0x7f64340ffeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f64337fe700 1 -- 172.21.15.88:0/2045094591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64340fe210 msgr2=0x7f64340ff3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f64337fe700 1 --2- 172.21.15.88:0/2045094591 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f64340fe210 0x7f64340ff3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f64337fe700 1 -- 172.21.15.88:0/2045094591 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64240095f0 con 0x7f64340febc0 2023-07-14T17:43:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f64337fe700 1 --2- 172.21.15.88:0/2045094591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64340febc0 0x7f64340ff930 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f642800d980 tx=0x7f642800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:28.560 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f64317fa700 1 -- 172.21.15.88:0/2045094591 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6428004220 con 0x7f64340febc0 2023-07-14T17:43:28.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f643a5d1700 1 -- 172.21.15.88:0/2045094591 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64341a46c0 con 0x7f64340febc0 2023-07-14T17:43:28.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f64317fa700 1 -- 172.21.15.88:0/2045094591 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6428007640 con 0x7f64340febc0 2023-07-14T17:43:28.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.918+0000 7f64317fa700 1 -- 172.21.15.88:0/2045094591 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6428005660 con 0x7f64340febc0 2023-07-14T17:43:28.561 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.919+0000 7f643a5d1700 1 -- 172.21.15.88:0/2045094591 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64341a4bc0 con 0x7f64340febc0 2023-07-14T17:43:28.563 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.919+0000 7f643a5d1700 1 -- 172.21.15.88:0/2045094591 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f643410fe10 con 0x7f64340febc0 2023-07-14T17:43:28.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.921+0000 7f64317fa700 1 -- 172.21.15.88:0/2045094591 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6428004750 con 0x7f64340febc0 2023-07-14T17:43:28.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.921+0000 7f64317fa700 1 --2- 172.21.15.88:0/2045094591 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f641c0a11e0 0x7f641c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.921+0000 7f6433fff700 1 --2- 172.21.15.88:0/2045094591 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f641c0a11e0 0x7f641c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.921+0000 7f64317fa700 1 -- 172.21.15.88:0/2045094591 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f64280cd0b0 con 0x7f64340febc0 2023-07-14T17:43:28.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.921+0000 7f64317fa700 1 --2- 172.21.15.88:0/2045094591 >> [v2:172.21.15.107:6824/1223599278,v1:172.21.15.107:6825/1223599278] conn(0x7f641c0abd50 0x7f641c0ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.564 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.922+0000 7f64317fa700 1 -- 172.21.15.88:0/2045094591 --> [v2:172.21.15.107:6824/1223599278,v1:172.21.15.107:6825/1223599278] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f641c0ae850 con 0x7f641c0abd50 2023-07-14T17:43:28.565 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.922+0000 7f64317fa700 1 -- 172.21.15.88:0/2045094591 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64280cd4c0 con 0x7f64340febc0 2023-07-14T17:43:28.566 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.922+0000 7f6433fff700 1 --2- 172.21.15.88:0/2045094591 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f641c0a11e0 0x7f641c0a3690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f642400b5c0 tx=0x7f6424005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:28.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.922+0000 7f6438b6e700 1 --2- 172.21.15.88:0/2045094591 >> [v2:172.21.15.107:6824/1223599278,v1:172.21.15.107:6825/1223599278] conn(0x7f641c0abd50 0x7f641c0ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.922+0000 7f6438b6e700 1 --2- 172.21.15.88:0/2045094591 >> [v2:172.21.15.107:6824/1223599278,v1:172.21.15.107:6825/1223599278] conn(0x7f641c0abd50 0x7f641c0ae180 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:28.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.923+0000 7f64317fa700 1 -- 172.21.15.88:0/2045094591 <== osd.10 v2:172.21.15.107:6824/1223599278 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f641c0ae850 con 0x7f641c0abd50 2023-07-14T17:43:28.567 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.940+0000 7f643a5d1700 1 -- 172.21.15.88:0/2045094591 --> [v2:172.21.15.107:6824/1223599278,v1:172.21.15.107:6825/1223599278] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f643404e340 con 0x7f641c0abd50 2023-07-14T17:43:28.568 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.940+0000 7f64317fa700 1 -- 172.21.15.88:0/2045094591 <== osd.10 v2:172.21.15.107:6824/1223599278 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f643404e340 con 0x7f641c0abd50 2023-07-14T17:43:28.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.941+0000 7f643a5d1700 1 -- 172.21.15.88:0/2045094591 >> [v2:172.21.15.107:6824/1223599278,v1:172.21.15.107:6825/1223599278] conn(0x7f641c0abd50 msgr2=0x7f641c0ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.569 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.941+0000 7f643a5d1700 1 --2- 172.21.15.88:0/2045094591 >> [v2:172.21.15.107:6824/1223599278,v1:172.21.15.107:6825/1223599278] conn(0x7f641c0abd50 0x7f641c0ae180 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.941+0000 7f643a5d1700 1 -- 172.21.15.88:0/2045094591 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f641c0a11e0 msgr2=0x7f641c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.941+0000 7f643a5d1700 1 --2- 172.21.15.88:0/2045094591 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f641c0a11e0 0x7f641c0a3690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f642400b5c0 tx=0x7f6424005fb0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.941+0000 7f643a5d1700 1 -- 172.21.15.88:0/2045094591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64340febc0 msgr2=0x7f64340ff930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.941+0000 7f643a5d1700 1 --2- 172.21.15.88:0/2045094591 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f64340febc0 0x7f64340ff930 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f642800d980 tx=0x7f642800dc90 comp rx=0 tx=0).stop 2023-07-14T17:43:28.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.941+0000 7f6438b6e700 1 -- 172.21.15.88:0/2045094591 reap_dead start 2023-07-14T17:43:28.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.941+0000 7f643a5d1700 1 -- 172.21.15.88:0/2045094591 shutdown_connections 2023-07-14T17:43:28.570 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.941+0000 7f643a5d1700 1 -- 172.21.15.88:0/2045094591 >> 172.21.15.88:0/2045094591 conn(0x7f6434075280 msgr2=0x7f64340fbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:28.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.941+0000 7f643a5d1700 1 -- 172.21.15.88:0/2045094591 shutdown_connections 2023-07-14T17:43:28.571 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:27.941+0000 7f643a5d1700 1 -- 172.21.15.88:0/2045094591 wait complete. 2023-07-14T17:43:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.442+0000 7fc752683700 1 -- 172.21.15.88:0/111566297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc74c1008d0 msgr2=0x7fc74c1048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.442+0000 7fc752683700 1 --2- 172.21.15.88:0/111566297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc74c1008d0 0x7fc74c1048a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc7400099c0 tx=0x7fc740009cd0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.443+0000 7fc752683700 1 -- 172.21.15.88:0/111566297 shutdown_connections 2023-07-14T17:43:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.443+0000 7fc752683700 1 --2- 172.21.15.88:0/111566297 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc74c104fd0 0x7fc74c10cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.443+0000 7fc752683700 1 --2- 172.21.15.88:0/111566297 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc74c0fff30 0x7fc74c100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.443+0000 7fc752683700 1 --2- 172.21.15.88:0/111566297 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc74c1008d0 0x7fc74c1048a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.443+0000 7fc752683700 1 -- 172.21.15.88:0/111566297 >> 172.21.15.88:0/111566297 conn(0x7fc74c0fb790 msgr2=0x7fc74c0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.443+0000 7fc752683700 1 -- 172.21.15.88:0/111566297 shutdown_connections 2023-07-14T17:43:28.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.443+0000 7fc752683700 1 -- 172.21.15.88:0/111566297 wait complete. 2023-07-14T17:43:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.443+0000 7fc752683700 1 Processor -- start 2023-07-14T17:43:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.443+0000 7fc752683700 1 -- start start 2023-07-14T17:43:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc752683700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc74c0fff30 0x7fc74c195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc752683700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc74c1008d0 0x7fc74c1962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc752683700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc74c104fd0 0x7fc74c19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc752683700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c19ad30 con 0x7fc74c0fff30 2023-07-14T17:43:28.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc752683700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c19aea0 con 0x7fc74c1008d0 2023-07-14T17:43:28.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc752683700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74c19b060 con 0x7fc74c104fd0 2023-07-14T17:43:28.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc74bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc74c0fff30 0x7fc74c195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc74bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc74c0fff30 0x7fc74c195d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42254/0 (socket says 172.21.15.88:42254) 2023-07-14T17:43:28.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc74bfff700 1 -- 172.21.15.88:0/552246544 learned_addr learned my addr 172.21.15.88:0/552246544 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:28.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc750c20700 1 --2- 172.21.15.88:0/552246544 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc74c104fd0 0x7fc74c19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc74b7fe700 1 --2- 172.21.15.88:0/552246544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc74c1008d0 0x7fc74c1962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc74bfff700 1 -- 172.21.15.88:0/552246544 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc74c104fd0 msgr2=0x7fc74c19a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc74bfff700 1 --2- 172.21.15.88:0/552246544 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc74c104fd0 0x7fc74c19a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc74bfff700 1 -- 172.21.15.88:0/552246544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc74c1008d0 msgr2=0x7fc74c1962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc74bfff700 1 --2- 172.21.15.88:0/552246544 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc74c1008d0 0x7fc74c1962b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc74bfff700 1 -- 172.21.15.88:0/552246544 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7400095f0 con 0x7fc74c0fff30 2023-07-14T17:43:28.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc74bfff700 1 --2- 172.21.15.88:0/552246544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc74c0fff30 0x7fc74c195d70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc73c00d950 tx=0x7fc73c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:28.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc7497fa700 1 -- 172.21.15.88:0/552246544 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73c009940 con 0x7fc74c0fff30 2023-07-14T17:43:28.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc7497fa700 1 -- 172.21.15.88:0/552246544 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc73c010460 con 0x7fc74c0fff30 2023-07-14T17:43:28.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc7497fa700 1 -- 172.21.15.88:0/552246544 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc73c00f620 con 0x7fc74c0fff30 2023-07-14T17:43:28.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc752683700 1 -- 172.21.15.88:0/552246544 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc74c19b340 con 0x7fc74c0fff30 2023-07-14T17:43:28.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.444+0000 7fc752683700 1 -- 172.21.15.88:0/552246544 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc74c1a2ba0 con 0x7fc74c0fff30 2023-07-14T17:43:28.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.445+0000 7fc752683700 1 -- 172.21.15.88:0/552246544 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc74c10fcf0 con 0x7fc74c0fff30 2023-07-14T17:43:28.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.446+0000 7fc7497fa700 1 -- 172.21.15.88:0/552246544 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc73c0105d0 con 0x7fc74c0fff30 2023-07-14T17:43:28.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.447+0000 7fc7497fa700 1 --2- 172.21.15.88:0/552246544 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc7340a11e0 0x7fc7340a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.447+0000 7fc7497fa700 1 -- 172.21.15.88:0/552246544 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc73c0cd010 con 0x7fc74c0fff30 2023-07-14T17:43:28.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.447+0000 7fc74b7fe700 1 --2- 172.21.15.88:0/552246544 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc7340a11e0 0x7fc7340a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.447+0000 7fc7497fa700 1 --2- 172.21.15.88:0/552246544 >> [v2:172.21.15.178:6816/2305567673,v1:172.21.15.178:6817/2305567673] conn(0x7fc7340abd50 0x7fc7340ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:28.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.447+0000 7fc7497fa700 1 -- 172.21.15.88:0/552246544 --> [v2:172.21.15.178:6816/2305567673,v1:172.21.15.178:6817/2305567673] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc7340ae850 con 0x7fc7340abd50 2023-07-14T17:43:28.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.447+0000 7fc7497fa700 1 -- 172.21.15.88:0/552246544 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc73c0cd420 con 0x7fc74c0fff30 2023-07-14T17:43:28.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.447+0000 7fc74b7fe700 1 --2- 172.21.15.88:0/552246544 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc7340a11e0 0x7fc7340a3690 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc740000c00 tx=0x7fc740005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:28.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.447+0000 7fc750c20700 1 --2- 172.21.15.88:0/552246544 >> [v2:172.21.15.178:6816/2305567673,v1:172.21.15.178:6817/2305567673] conn(0x7fc7340abd50 0x7fc7340ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:28.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.448+0000 7fc750c20700 1 --2- 172.21.15.88:0/552246544 >> [v2:172.21.15.178:6816/2305567673,v1:172.21.15.178:6817/2305567673] conn(0x7fc7340abd50 0x7fc7340ae180 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:28.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.449+0000 7fc7497fa700 1 -- 172.21.15.88:0/552246544 <== osd.8 v2:172.21.15.178:6816/2305567673 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc7340ae850 con 0x7fc7340abd50 2023-07-14T17:43:28.820 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.464+0000 7fc752683700 1 -- 172.21.15.88:0/552246544 --> [v2:172.21.15.178:6816/2305567673,v1:172.21.15.178:6817/2305567673] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc74c04e340 con 0x7fc7340abd50 2023-07-14T17:43:28.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.465+0000 7fc7497fa700 1 -- 172.21.15.88:0/552246544 <== osd.8 v2:172.21.15.178:6816/2305567673 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc74c04e340 con 0x7fc7340abd50 2023-07-14T17:43:28.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.465+0000 7fc752683700 1 -- 172.21.15.88:0/552246544 >> [v2:172.21.15.178:6816/2305567673,v1:172.21.15.178:6817/2305567673] conn(0x7fc7340abd50 msgr2=0x7fc7340ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.465+0000 7fc752683700 1 --2- 172.21.15.88:0/552246544 >> [v2:172.21.15.178:6816/2305567673,v1:172.21.15.178:6817/2305567673] conn(0x7fc7340abd50 0x7fc7340ae180 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.465+0000 7fc752683700 1 -- 172.21.15.88:0/552246544 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc7340a11e0 msgr2=0x7fc7340a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.465+0000 7fc752683700 1 --2- 172.21.15.88:0/552246544 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc7340a11e0 0x7fc7340a3690 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc740000c00 tx=0x7fc740005fb0 comp rx=0 tx=0).stop 2023-07-14T17:43:28.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.465+0000 7fc752683700 1 -- 172.21.15.88:0/552246544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc74c0fff30 msgr2=0x7fc74c195d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:28.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.465+0000 7fc752683700 1 --2- 172.21.15.88:0/552246544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc74c0fff30 0x7fc74c195d70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc73c00d950 tx=0x7fc73c00dc60 comp rx=0 tx=0).stop 2023-07-14T17:43:28.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.465+0000 7fc750c20700 1 -- 172.21.15.88:0/552246544 reap_dead start 2023-07-14T17:43:28.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.465+0000 7fc752683700 1 -- 172.21.15.88:0/552246544 shutdown_connections 2023-07-14T17:43:28.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.465+0000 7fc752683700 1 -- 172.21.15.88:0/552246544 >> 172.21.15.88:0/552246544 conn(0x7fc74c0fb790 msgr2=0x7fc74c0fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:28.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.465+0000 7fc752683700 1 -- 172.21.15.88:0/552246544 shutdown_connections 2023-07-14T17:43:28.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:28.466+0000 7fc752683700 1 -- 172.21.15.88:0/552246544 wait complete. 2023-07-14T17:43:28.844 INFO:teuthology.orchestra.run.smithi088.stdout:55834574911 2023-07-14T17:43:28.845 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.1 2023-07-14T17:43:28.958 INFO:teuthology.orchestra.run.smithi088.stdout:34359738435 2023-07-14T17:43:28.958 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.0 2023-07-14T17:43:29.167 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:28 smithi088 ceph-mon[127465]: pgmap v278: 129 pgs: 129 active+clean; 639 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2023-07-14T17:43:29.168 INFO:teuthology.orchestra.run.smithi088.stdout:107374182454 2023-07-14T17:43:29.169 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.3 2023-07-14T17:43:29.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:28 smithi107 ceph-mon[128999]: pgmap v278: 129 pgs: 129 active+clean; 639 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2023-07-14T17:43:29.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:28 smithi178 ceph-mon[129368]: pgmap v278: 129 pgs: 129 active+clean; 639 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2023-07-14T17:43:29.476 INFO:teuthology.orchestra.run.smithi088.stdout:244813135905 2023-07-14T17:43:29.476 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.8 2023-07-14T17:43:29.673 INFO:teuthology.orchestra.run.smithi088.stdout:296352743448 2023-07-14T17:43:29.674 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.10 2023-07-14T17:43:29.681 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:29.820 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:29.845 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:30.309 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:30.310 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:30.310 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:30.482 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:30.484 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:30.488 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:31.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:30 smithi088 ceph-mon[127465]: pgmap v279: 129 pgs: 129 active+clean; 639 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 10 op/s 2023-07-14T17:43:31.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:30 smithi107 ceph-mon[128999]: pgmap v279: 129 pgs: 129 active+clean; 639 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 10 op/s 2023-07-14T17:43:31.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:30 smithi178 ceph-mon[129368]: pgmap v279: 129 pgs: 129 active+clean; 639 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 10 op/s 2023-07-14T17:43:32.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.534+0000 7f471e585700 1 -- 172.21.15.88:0/1507702448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47180fefc0 msgr2=0x7f4718102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:32.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.534+0000 7f471e585700 1 --2- 172.21.15.88:0/1507702448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47180fefc0 0x7f4718102f90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f470c009990 tx=0x7f470c009ca0 comp rx=0 tx=0).stop 2023-07-14T17:43:32.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.534+0000 7f471e585700 1 -- 172.21.15.88:0/1507702448 shutdown_connections 2023-07-14T17:43:32.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.534+0000 7f471e585700 1 --2- 172.21.15.88:0/1507702448 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47181036c0 0x7f471810b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:32.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.534+0000 7f471e585700 1 --2- 172.21.15.88:0/1507702448 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f47180fe620 0x7f47180fe9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:32.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.534+0000 7f471e585700 1 --2- 172.21.15.88:0/1507702448 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47180fefc0 0x7f4718102f90 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:32.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.534+0000 7f471e585700 1 -- 172.21.15.88:0/1507702448 >> 172.21.15.88:0/1507702448 conn(0x7f47180751e0 msgr2=0x7f47180755e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:32.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.534+0000 7f471e585700 1 -- 172.21.15.88:0/1507702448 shutdown_connections 2023-07-14T17:43:32.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.534+0000 7f471e585700 1 -- 172.21.15.88:0/1507702448 wait complete. 2023-07-14T17:43:32.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.535+0000 7f471e585700 1 Processor -- start 2023-07-14T17:43:32.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.535+0000 7f471e585700 1 -- start start 2023-07-14T17:43:32.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.535+0000 7f471e585700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47180fe620 0x7f471819a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:32.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.535+0000 7f471e585700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f47180fefc0 0x7f471819aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:32.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.535+0000 7f471e585700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47181036c0 0x7f4718194a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:32.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.535+0000 7f471e585700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4718195090 con 0x7f47181036c0 2023-07-14T17:43:32.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.535+0000 7f471e585700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4718195200 con 0x7f47180fe620 2023-07-14T17:43:32.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.535+0000 7f471e585700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47181953c0 con 0x7f47180fefc0 2023-07-14T17:43:32.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.536+0000 7f471cb22700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47181036c0 0x7f4718194a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:32.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.536+0000 7f471cb22700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47181036c0 0x7f4718194a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42274/0 (socket says 172.21.15.88:42274) 2023-07-14T17:43:32.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.536+0000 7f471cb22700 1 -- 172.21.15.88:0/3439803655 learned_addr learned my addr 172.21.15.88:0/3439803655 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:32.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.536+0000 7f47177fe700 1 --2- 172.21.15.88:0/3439803655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f47180fefc0 0x7f471819aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:32.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.536+0000 7f4717fff700 1 --2- 172.21.15.88:0/3439803655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47180fe620 0x7f471819a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:32.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.536+0000 7f471cb22700 1 -- 172.21.15.88:0/3439803655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f47180fefc0 msgr2=0x7f471819aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:32.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.536+0000 7f471cb22700 1 --2- 172.21.15.88:0/3439803655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f47180fefc0 0x7f471819aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:32.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.536+0000 7f471cb22700 1 -- 172.21.15.88:0/3439803655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47180fe620 msgr2=0x7f471819a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:32.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.536+0000 7f471cb22700 1 --2- 172.21.15.88:0/3439803655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47180fe620 0x7f471819a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:32.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.536+0000 7f471cb22700 1 -- 172.21.15.88:0/3439803655 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f470c0095f0 con 0x7f47181036c0 2023-07-14T17:43:32.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.536+0000 7f471cb22700 1 --2- 172.21.15.88:0/3439803655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47181036c0 0x7f4718194a20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f471000e580 tx=0x7f471000e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:32.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.536+0000 7f47157fa700 1 -- 172.21.15.88:0/3439803655 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4710004db0 con 0x7f47181036c0 2023-07-14T17:43:32.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.536+0000 7f471e585700 1 -- 172.21.15.88:0/3439803655 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47181956a0 con 0x7f47181036c0 2023-07-14T17:43:32.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.536+0000 7f47157fa700 1 -- 172.21.15.88:0/3439803655 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4710003e90 con 0x7f47181036c0 2023-07-14T17:43:32.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.536+0000 7f47157fa700 1 -- 172.21.15.88:0/3439803655 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4710004490 con 0x7f47181036c0 2023-07-14T17:43:32.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.536+0000 7f471e585700 1 -- 172.21.15.88:0/3439803655 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471810b950 con 0x7f47181036c0 2023-07-14T17:43:32.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.537+0000 7f471e585700 1 -- 172.21.15.88:0/3439803655 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471804e340 con 0x7f47181036c0 2023-07-14T17:43:32.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.540+0000 7f47157fa700 1 -- 172.21.15.88:0/3439803655 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4710009090 con 0x7f47181036c0 2023-07-14T17:43:32.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.540+0000 7f47157fa700 1 --2- 172.21.15.88:0/3439803655 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f47000a11e0 0x7f47000a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:32.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.541+0000 7f4717fff700 1 --2- 172.21.15.88:0/3439803655 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f47000a11e0 0x7f47000a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:32.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.541+0000 7f47157fa700 1 -- 172.21.15.88:0/3439803655 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4710016070 con 0x7f47181036c0 2023-07-14T17:43:32.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.541+0000 7f4717fff700 1 --2- 172.21.15.88:0/3439803655 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f47000a11e0 0x7f47000a3690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4708009c60 tx=0x7f4708009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:32.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.543+0000 7f47157fa700 1 -- 172.21.15.88:0/3439803655 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4710098340 con 0x7f47181036c0 2023-07-14T17:43:32.723 INFO:teuthology.orchestra.run.smithi088.stdout:137438953522 2023-07-14T17:43:32.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.740+0000 7f471e585700 1 -- 172.21.15.88:0/3439803655 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4718195f30 con 0x7f47181036c0 2023-07-14T17:43:32.724 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.740+0000 7f47157fa700 1 -- 172.21.15.88:0/3439803655 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f471009b960 con 0x7f47181036c0 2023-07-14T17:43:32.725 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.743+0000 7f471e585700 1 -- 172.21.15.88:0/3439803655 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f47000a11e0 msgr2=0x7f47000a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.743+0000 7f471e585700 1 --2- 172.21.15.88:0/3439803655 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f47000a11e0 0x7f47000a3690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4708009c60 tx=0x7f4708009260 comp rx=0 tx=0).stop 2023-07-14T17:43:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.743+0000 7f471e585700 1 -- 172.21.15.88:0/3439803655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47181036c0 msgr2=0x7f4718194a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.743+0000 7f471e585700 1 --2- 172.21.15.88:0/3439803655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47181036c0 0x7f4718194a20 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f471000e580 tx=0x7f471000e890 comp rx=0 tx=0).stop 2023-07-14T17:43:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.743+0000 7f471e585700 1 -- 172.21.15.88:0/3439803655 shutdown_connections 2023-07-14T17:43:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.743+0000 7f471e585700 1 --2- 172.21.15.88:0/3439803655 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f47000a11e0 0x7f47000a3690 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.743+0000 7f471e585700 1 --2- 172.21.15.88:0/3439803655 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47181036c0 0x7f4718194a20 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.743+0000 7f471e585700 1 --2- 172.21.15.88:0/3439803655 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47180fe620 0x7f471819a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.743+0000 7f471e585700 1 --2- 172.21.15.88:0/3439803655 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f47180fefc0 0x7f471819aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.743+0000 7f471e585700 1 -- 172.21.15.88:0/3439803655 >> 172.21.15.88:0/3439803655 conn(0x7f47180751e0 msgr2=0x7f47180fbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.743+0000 7f471e585700 1 -- 172.21.15.88:0/3439803655 shutdown_connections 2023-07-14T17:43:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:32.743+0000 7f471e585700 1 -- 172.21.15.88:0/3439803655 wait complete. 2023-07-14T17:43:32.997 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:32 smithi088 ceph-mon[127465]: pgmap v280: 129 pgs: 129 active+clean; 639 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2023-07-14T17:43:32.997 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:32 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3439803655' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T17:43:33.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:32 smithi107 ceph-mon[128999]: pgmap v280: 129 pgs: 129 active+clean; 639 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2023-07-14T17:43:33.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:32 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3439803655' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T17:43:33.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:32 smithi178 ceph-mon[129368]: pgmap v280: 129 pgs: 129 active+clean; 639 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2023-07-14T17:43:33.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:32 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3439803655' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T17:43:33.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.959+0000 7f470c4a5700 1 -- 172.21.15.88:0/1193058394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4704104fd0 msgr2=0x7f470410cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:33.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.959+0000 7f470c4a5700 1 --2- 172.21.15.88:0/1193058394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4704104fd0 0x7f470410cb50 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f46fc00b0d0 tx=0x7f46fc00b3e0 comp rx=0 tx=0).stop 2023-07-14T17:43:33.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.959+0000 7f470c4a5700 1 -- 172.21.15.88:0/1193058394 shutdown_connections 2023-07-14T17:43:33.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.959+0000 7f470c4a5700 1 --2- 172.21.15.88:0/1193058394 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4704104fd0 0x7f470410cb50 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:33.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.959+0000 7f470c4a5700 1 --2- 172.21.15.88:0/1193058394 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f47040fff30 0x7f4704100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:33.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.959+0000 7f470c4a5700 1 --2- 172.21.15.88:0/1193058394 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47041008d0 0x7f47041048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:33.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.959+0000 7f470c4a5700 1 -- 172.21.15.88:0/1193058394 >> 172.21.15.88:0/1193058394 conn(0x7f47040fb790 msgr2=0x7f47040fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:33.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.959+0000 7f470c4a5700 1 -- 172.21.15.88:0/1193058394 shutdown_connections 2023-07-14T17:43:33.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.959+0000 7f470c4a5700 1 -- 172.21.15.88:0/1193058394 wait complete. 2023-07-14T17:43:33.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.960+0000 7f470c4a5700 1 Processor -- start 2023-07-14T17:43:33.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.960+0000 7f470c4a5700 1 -- start start 2023-07-14T17:43:33.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.960+0000 7f470c4a5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47040fff30 0x7f4704072940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:33.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.960+0000 7f470c4a5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47041008d0 0x7f4704072e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.960+0000 7f470c4a5700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4704104fd0 0x7f4704073400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.960+0000 7f470c4a5700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47041a42a0 con 0x7f47040fff30 2023-07-14T17:43:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.960+0000 7f470c4a5700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47041a4410 con 0x7f47041008d0 2023-07-14T17:43:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.960+0000 7f470c4a5700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47041a45d0 con 0x7f4704104fd0 2023-07-14T17:43:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.960+0000 7f470a241700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47040fff30 0x7f4704072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.960+0000 7f470a241700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47040fff30 0x7f4704072940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42282/0 (socket says 172.21.15.88:42282) 2023-07-14T17:43:33.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.960+0000 7f470a241700 1 -- 172.21.15.88:0/3387107540 learned_addr learned my addr 172.21.15.88:0/3387107540 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:33.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.960+0000 7f4709a40700 1 --2- 172.21.15.88:0/3387107540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47041008d0 0x7f4704072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:33.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.961+0000 7f470aa42700 1 --2- 172.21.15.88:0/3387107540 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4704104fd0 0x7f4704073400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:33.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.961+0000 7f470a241700 1 -- 172.21.15.88:0/3387107540 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4704104fd0 msgr2=0x7f4704073400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:33.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.961+0000 7f470a241700 1 --2- 172.21.15.88:0/3387107540 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4704104fd0 0x7f4704073400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:33.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.961+0000 7f470a241700 1 -- 172.21.15.88:0/3387107540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47041008d0 msgr2=0x7f4704072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:33.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.961+0000 7f470a241700 1 --2- 172.21.15.88:0/3387107540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47041008d0 0x7f4704072e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:33.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.961+0000 7f470a241700 1 -- 172.21.15.88:0/3387107540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc009d00 con 0x7f47040fff30 2023-07-14T17:43:33.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.961+0000 7f470a241700 1 --2- 172.21.15.88:0/3387107540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47040fff30 0x7f4704072940 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f470000c920 tx=0x7f470000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:33.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.961+0000 7f46fb7fe700 1 -- 172.21.15.88:0/3387107540 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4700007b00 con 0x7f47040fff30 2023-07-14T17:43:33.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.961+0000 7f470c4a5700 1 -- 172.21.15.88:0/3387107540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47041a48b0 con 0x7f47040fff30 2023-07-14T17:43:33.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.961+0000 7f46fb7fe700 1 -- 172.21.15.88:0/3387107540 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f470000ce70 con 0x7f47040fff30 2023-07-14T17:43:33.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.961+0000 7f46fb7fe700 1 -- 172.21.15.88:0/3387107540 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47000053e0 con 0x7f47040fff30 2023-07-14T17:43:33.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.961+0000 7f470c4a5700 1 -- 172.21.15.88:0/3387107540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47041a4db0 con 0x7f47040fff30 2023-07-14T17:43:33.948 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.962+0000 7f470c4a5700 1 -- 172.21.15.88:0/3387107540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f470404e340 con 0x7f47040fff30 2023-07-14T17:43:33.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.963+0000 7f46fb7fe700 1 -- 172.21.15.88:0/3387107540 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4700005560 con 0x7f47040fff30 2023-07-14T17:43:33.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.964+0000 7f46fb7fe700 1 --2- 172.21.15.88:0/3387107540 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f46f00a1260 0x7f46f00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:33.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.964+0000 7f4709a40700 1 --2- 172.21.15.88:0/3387107540 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f46f00a1260 0x7f46f00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:33.949 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.964+0000 7f46fb7fe700 1 -- 172.21.15.88:0/3387107540 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f47000ccce0 con 0x7f47040fff30 2023-07-14T17:43:33.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.964+0000 7f4709a40700 1 --2- 172.21.15.88:0/3387107540 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f46f00a1260 0x7f46f00a3710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4704074130 tx=0x7f46f400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:33.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:33.967+0000 7f46fb7fe700 1 -- 172.21.15.88:0/3387107540 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4700096080 con 0x7f47040fff30 2023-07-14T17:43:34.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.170+0000 7f470c4a5700 1 -- 172.21.15.88:0/3387107540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f47041a5100 con 0x7f47040fff30 2023-07-14T17:43:34.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.170+0000 7f46fb7fe700 1 -- 172.21.15.88:0/3387107540 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f470001d020 con 0x7f47040fff30 2023-07-14T17:43:34.154 INFO:teuthology.orchestra.run.smithi088.stdout:210453397542 2023-07-14T17:43:34.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.173+0000 7f470c4a5700 1 -- 172.21.15.88:0/3387107540 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f46f00a1260 msgr2=0x7f46f00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:34.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.173+0000 7f470c4a5700 1 --2- 172.21.15.88:0/3387107540 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f46f00a1260 0x7f46f00a3710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f4704074130 tx=0x7f46f400b410 comp rx=0 tx=0).stop 2023-07-14T17:43:34.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.173+0000 7f470c4a5700 1 -- 172.21.15.88:0/3387107540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47040fff30 msgr2=0x7f4704072940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:34.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.173+0000 7f470c4a5700 1 --2- 172.21.15.88:0/3387107540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47040fff30 0x7f4704072940 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f470000c920 tx=0x7f470000cc30 comp rx=0 tx=0).stop 2023-07-14T17:43:34.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.173+0000 7f470c4a5700 1 -- 172.21.15.88:0/3387107540 shutdown_connections 2023-07-14T17:43:34.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.173+0000 7f470c4a5700 1 --2- 172.21.15.88:0/3387107540 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4704104fd0 0x7f4704073400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:34.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.173+0000 7f470c4a5700 1 --2- 172.21.15.88:0/3387107540 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f46f00a1260 0x7f46f00a3710 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:34.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.173+0000 7f470c4a5700 1 --2- 172.21.15.88:0/3387107540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f47040fff30 0x7f4704072940 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:34.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.173+0000 7f470c4a5700 1 --2- 172.21.15.88:0/3387107540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f47041008d0 0x7f4704072e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:34.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.173+0000 7f470c4a5700 1 -- 172.21.15.88:0/3387107540 >> 172.21.15.88:0/3387107540 conn(0x7f47040fb790 msgr2=0x7f47040fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:34.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.173+0000 7f470c4a5700 1 -- 172.21.15.88:0/3387107540 shutdown_connections 2023-07-14T17:43:34.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.173+0000 7f470c4a5700 1 -- 172.21.15.88:0/3387107540 wait complete. 2023-07-14T17:43:34.189 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:34.259 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:34.290 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953520 got 137438953522 for osd.4 2023-07-14T17:43:34.290 DEBUG:teuthology.parallel:result is None 2023-07-14T17:43:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:34.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.727+0000 7fb227a4f700 1 -- 172.21.15.88:0/1562430112 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2200fe5e0 msgr2=0x7fb2200fe9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.727+0000 7fb227a4f700 1 --2- 172.21.15.88:0/1562430112 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2200fe5e0 0x7fb2200fe9b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb21c0099c0 tx=0x7fb21c009cd0 comp rx=0 tx=0).stop 2023-07-14T17:43:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.727+0000 7fb227a4f700 1 -- 172.21.15.88:0/1562430112 shutdown_connections 2023-07-14T17:43:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.727+0000 7fb227a4f700 1 --2- 172.21.15.88:0/1562430112 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb220103680 0x7fb22010b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.727+0000 7fb227a4f700 1 --2- 172.21.15.88:0/1562430112 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2200fe5e0 0x7fb2200fe9b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.727+0000 7fb227a4f700 1 --2- 172.21.15.88:0/1562430112 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2200fef80 0x7fb220102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.727+0000 7fb227a4f700 1 -- 172.21.15.88:0/1562430112 >> 172.21.15.88:0/1562430112 conn(0x7fb220075280 msgr2=0x7fb220075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.727+0000 7fb227a4f700 1 -- 172.21.15.88:0/1562430112 shutdown_connections 2023-07-14T17:43:34.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.727+0000 7fb227a4f700 1 -- 172.21.15.88:0/1562430112 wait complete. 2023-07-14T17:43:34.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb227a4f700 1 Processor -- start 2023-07-14T17:43:34.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb227a4f700 1 -- start start 2023-07-14T17:43:34.715 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb227a4f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2200fe5e0 0x7fb220196b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:34.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb227a4f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2200fef80 0x7fb220197050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:34.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb227a4f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb220103680 0x7fb2201a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:34.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb227a4f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2201a20d0 con 0x7fb2200fef80 2023-07-14T17:43:34.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb227a4f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2201a2210 con 0x7fb2200fe5e0 2023-07-14T17:43:34.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb227a4f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2201a2350 con 0x7fb220103680 2023-07-14T17:43:34.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb224fea700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2200fef80 0x7fb220197050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:34.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb224fea700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2200fef80 0x7fb220197050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42298/0 (socket says 172.21.15.88:42298) 2023-07-14T17:43:34.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb224fea700 1 -- 172.21.15.88:0/3594396299 learned_addr learned my addr 172.21.15.88:0/3594396299 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:34.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb2257eb700 1 --2- 172.21.15.88:0/3594396299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2200fe5e0 0x7fb220196b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:34.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb225fec700 1 --2- 172.21.15.88:0/3594396299 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb220103680 0x7fb2201a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:34.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb224fea700 1 -- 172.21.15.88:0/3594396299 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb220103680 msgr2=0x7fb2201a1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:34.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb224fea700 1 --2- 172.21.15.88:0/3594396299 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb220103680 0x7fb2201a1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:34.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb224fea700 1 -- 172.21.15.88:0/3594396299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2200fe5e0 msgr2=0x7fb220196b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:34.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb224fea700 1 --2- 172.21.15.88:0/3594396299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2200fe5e0 0x7fb220196b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:34.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb224fea700 1 -- 172.21.15.88:0/3594396299 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21c0095f0 con 0x7fb2200fef80 2023-07-14T17:43:34.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.728+0000 7fb225fec700 1 --2- 172.21.15.88:0/3594396299 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb220103680 0x7fb2201a1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:43:34.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.729+0000 7fb224fea700 1 --2- 172.21.15.88:0/3594396299 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2200fef80 0x7fb220197050 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb21000e970 tx=0x7fb21000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:34.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.729+0000 7fb2167fc700 1 -- 172.21.15.88:0/3594396299 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb210009790 con 0x7fb2200fef80 2023-07-14T17:43:34.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.729+0000 7fb227a4f700 1 -- 172.21.15.88:0/3594396299 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2201a25b0 con 0x7fb2200fef80 2023-07-14T17:43:34.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.729+0000 7fb2167fc700 1 -- 172.21.15.88:0/3594396299 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb210004cc0 con 0x7fb2200fef80 2023-07-14T17:43:34.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.729+0000 7fb227a4f700 1 -- 172.21.15.88:0/3594396299 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2201a2b00 con 0x7fb2200fef80 2023-07-14T17:43:34.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.729+0000 7fb2167fc700 1 -- 172.21.15.88:0/3594396299 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2100055b0 con 0x7fb2200fef80 2023-07-14T17:43:34.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.730+0000 7fb227a4f700 1 -- 172.21.15.88:0/3594396299 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb22004e340 con 0x7fb2200fef80 2023-07-14T17:43:34.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.731+0000 7fb2167fc700 1 -- 172.21.15.88:0/3594396299 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb210004750 con 0x7fb2200fef80 2023-07-14T17:43:34.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.731+0000 7fb2167fc700 1 --2- 172.21.15.88:0/3594396299 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb20c0a1260 0x7fb20c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:34.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.731+0000 7fb2257eb700 1 --2- 172.21.15.88:0/3594396299 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb20c0a1260 0x7fb20c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:34.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.731+0000 7fb2167fc700 1 -- 172.21.15.88:0/3594396299 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb210014070 con 0x7fb2200fef80 2023-07-14T17:43:34.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.732+0000 7fb2257eb700 1 --2- 172.21.15.88:0/3594396299 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb20c0a1260 0x7fb20c0a3710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00b580 tx=0x7fb21c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:34.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.735+0000 7fb2167fc700 1 -- 172.21.15.88:0/3594396299 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb210096300 con 0x7fb2200fef80 2023-07-14T17:43:34.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.934+0000 7fb227a4f700 1 -- 172.21.15.88:0/3594396299 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fb2201a2dd0 con 0x7fb2200fef80 2023-07-14T17:43:34.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.934+0000 7fb2167fc700 1 -- 172.21.15.88:0/3594396299 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb210099920 con 0x7fb2200fef80 2023-07-14T17:43:34.918 INFO:teuthology.orchestra.run.smithi088.stdout:188978561066 2023-07-14T17:43:34.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.936+0000 7fb227a4f700 1 -- 172.21.15.88:0/3594396299 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb20c0a1260 msgr2=0x7fb20c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:34.919 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.936+0000 7fb227a4f700 1 --2- 172.21.15.88:0/3594396299 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb20c0a1260 0x7fb20c0a3710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00b580 tx=0x7fb21c01a040 comp rx=0 tx=0).stop 2023-07-14T17:43:34.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.936+0000 7fb227a4f700 1 -- 172.21.15.88:0/3594396299 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2200fef80 msgr2=0x7fb220197050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:34.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.936+0000 7fb227a4f700 1 --2- 172.21.15.88:0/3594396299 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2200fef80 0x7fb220197050 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb21000e970 tx=0x7fb21000ed30 comp rx=0 tx=0).stop 2023-07-14T17:43:34.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.936+0000 7fb227a4f700 1 -- 172.21.15.88:0/3594396299 shutdown_connections 2023-07-14T17:43:34.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.936+0000 7fb227a4f700 1 --2- 172.21.15.88:0/3594396299 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb220103680 0x7fb2201a1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:34.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.936+0000 7fb227a4f700 1 --2- 172.21.15.88:0/3594396299 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb2200fe5e0 0x7fb220196b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:34.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.936+0000 7fb227a4f700 1 --2- 172.21.15.88:0/3594396299 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb20c0a1260 0x7fb20c0a3710 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:34.920 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.937+0000 7fb227a4f700 1 --2- 172.21.15.88:0/3594396299 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb2200fef80 0x7fb220197050 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:34.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.937+0000 7fb227a4f700 1 -- 172.21.15.88:0/3594396299 >> 172.21.15.88:0/3594396299 conn(0x7fb220075280 msgr2=0x7fb2200fbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:34.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.937+0000 7fb227a4f700 1 -- 172.21.15.88:0/3594396299 shutdown_connections 2023-07-14T17:43:34.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:34.937+0000 7fb227a4f700 1 -- 172.21.15.88:0/3594396299 wait complete. 2023-07-14T17:43:35.041 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:34 smithi088 ceph-mon[127465]: pgmap v281: 129 pgs: 129 active+clean; 639 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2023-07-14T17:43:35.041 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:34 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3387107540' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T17:43:35.042 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:34 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3594396299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T17:43:35.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:34 smithi107 ceph-mon[128999]: pgmap v281: 129 pgs: 129 active+clean; 639 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2023-07-14T17:43:35.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:34 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3387107540' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T17:43:35.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:34 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3594396299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T17:43:35.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:34 smithi178 ceph-mon[129368]: pgmap v281: 129 pgs: 129 active+clean; 639 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 426 B/s wr, 9 op/s 2023-07-14T17:43:35.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:34 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3387107540' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T17:43:35.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:34 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3594396299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T17:43:35.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.932+0000 7f08fc4a9700 1 -- 172.21.15.88:0/1471159999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08f40febc0 msgr2=0x7f08f410cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:35.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.932+0000 7f08fc4a9700 1 --2- 172.21.15.88:0/1471159999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08f40febc0 0x7f08f410cb30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f08ec00b0a0 tx=0x7f08ec00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:43:35.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.932+0000 7f08fc4a9700 1 -- 172.21.15.88:0/1471159999 shutdown_connections 2023-07-14T17:43:35.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.932+0000 7f08fc4a9700 1 --2- 172.21.15.88:0/1471159999 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08f40febc0 0x7f08f410cb30 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:35.916 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.932+0000 7f08fc4a9700 1 --2- 172.21.15.88:0/1471159999 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08f4104c70 0x7f08f4105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:35.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.932+0000 7f08fc4a9700 1 --2- 172.21.15.88:0/1471159999 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f40fe210 0x7f08f40fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:35.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.932+0000 7f08fc4a9700 1 -- 172.21.15.88:0/1471159999 >> 172.21.15.88:0/1471159999 conn(0x7f08f4075280 msgr2=0x7f08f4075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:35.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.932+0000 7f08fc4a9700 1 -- 172.21.15.88:0/1471159999 shutdown_connections 2023-07-14T17:43:35.917 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.932+0000 7f08fc4a9700 1 -- 172.21.15.88:0/1471159999 wait complete. 2023-07-14T17:43:35.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.933+0000 7f08fc4a9700 1 Processor -- start 2023-07-14T17:43:35.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.933+0000 7f08fc4a9700 1 -- start start 2023-07-14T17:43:35.921 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.933+0000 7f08fc4a9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08f40fe210 0x7f08f40ff3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:35.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.933+0000 7f08fc4a9700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08f40febc0 0x7f08f40ff930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:35.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.933+0000 7f08fc4a9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f4104c70 0x7f08f41a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:35.922 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.933+0000 7f08fc4a9700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f4100220 con 0x7f08f40fe210 2023-07-14T17:43:35.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.933+0000 7f08fc4a9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f41a4490 con 0x7f08f4104c70 2023-07-14T17:43:35.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.933+0000 7f08fc4a9700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f41a4600 con 0x7f08f40febc0 2023-07-14T17:43:35.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.933+0000 7f08fa245700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08f40fe210 0x7f08f40ff3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:35.923 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.933+0000 7f08fa245700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08f40fe210 0x7f08f40ff3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42310/0 (socket says 172.21.15.88:42310) 2023-07-14T17:43:35.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.933+0000 7f08fa245700 1 -- 172.21.15.88:0/3245294107 learned_addr learned my addr 172.21.15.88:0/3245294107 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:35.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.933+0000 7f08faa46700 1 --2- 172.21.15.88:0/3245294107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f4104c70 0x7f08f41a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:35.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.933+0000 7f08f9a44700 1 --2- 172.21.15.88:0/3245294107 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08f40febc0 0x7f08f40ff930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:35.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.934+0000 7f08fa245700 1 -- 172.21.15.88:0/3245294107 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08f40febc0 msgr2=0x7f08f40ff930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:35.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.934+0000 7f08fa245700 1 --2- 172.21.15.88:0/3245294107 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08f40febc0 0x7f08f40ff930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:35.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.934+0000 7f08fa245700 1 -- 172.21.15.88:0/3245294107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f4104c70 msgr2=0x7f08f41a3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:35.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.934+0000 7f08fa245700 1 --2- 172.21.15.88:0/3245294107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f4104c70 0x7f08f41a3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:35.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.934+0000 7f08fa245700 1 -- 172.21.15.88:0/3245294107 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec009d00 con 0x7f08f40fe210 2023-07-14T17:43:35.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.934+0000 7f08fa245700 1 --2- 172.21.15.88:0/3245294107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08f40fe210 0x7f08f40ff3f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f08f000c9b0 tx=0x7f08f000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:35.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.934+0000 7f08eb7fe700 1 -- 172.21.15.88:0/3245294107 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f0007b00 con 0x7f08f40fe210 2023-07-14T17:43:35.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.934+0000 7f08eb7fe700 1 -- 172.21.15.88:0/3245294107 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f08f0007c60 con 0x7f08f40fe210 2023-07-14T17:43:35.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.934+0000 7f08fc4a9700 1 -- 172.21.15.88:0/3245294107 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f41a4880 con 0x7f08f40fe210 2023-07-14T17:43:35.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.934+0000 7f08eb7fe700 1 -- 172.21.15.88:0/3245294107 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f0018660 con 0x7f08f40fe210 2023-07-14T17:43:35.927 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.934+0000 7f08fc4a9700 1 -- 172.21.15.88:0/3245294107 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f41a4d80 con 0x7f08f40fe210 2023-07-14T17:43:35.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.935+0000 7f08fc4a9700 1 -- 172.21.15.88:0/3245294107 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f404e340 con 0x7f08f40fe210 2023-07-14T17:43:35.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.936+0000 7f08eb7fe700 1 -- 172.21.15.88:0/3245294107 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f08f000f450 con 0x7f08f40fe210 2023-07-14T17:43:35.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.937+0000 7f08eb7fe700 1 --2- 172.21.15.88:0/3245294107 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f08e00a11e0 0x7f08e00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:35.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.937+0000 7f08f9a44700 1 --2- 172.21.15.88:0/3245294107 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f08e00a11e0 0x7f08e00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:35.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.937+0000 7f08eb7fe700 1 -- 172.21.15.88:0/3245294107 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f08f00ccc10 con 0x7f08f40fe210 2023-07-14T17:43:35.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.938+0000 7f08f9a44700 1 --2- 172.21.15.88:0/3245294107 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f08e00a11e0 0x7f08e00a3690 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f08e4005950 tx=0x7f08e40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:35.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:35.941+0000 7f08eb7fe700 1 -- 172.21.15.88:0/3245294107 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f08f0096010 con 0x7f08f40fe210 2023-07-14T17:43:35.930 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397540 got 210453397542 for osd.7 2023-07-14T17:43:35.930 DEBUG:teuthology.parallel:result is None 2023-07-14T17:43:36.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.148+0000 7f08fc4a9700 1 -- 172.21.15.88:0/3245294107 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f08f4100920 con 0x7f08f40fe210 2023-07-14T17:43:36.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.148+0000 7f08eb7fe700 1 -- 172.21.15.88:0/3245294107 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f08f000fe60 con 0x7f08f40fe210 2023-07-14T17:43:36.132 INFO:teuthology.orchestra.run.smithi088.stdout:330712481814 2023-07-14T17:43:36.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.151+0000 7f08fc4a9700 1 -- 172.21.15.88:0/3245294107 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f08e00a11e0 msgr2=0x7f08e00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:36.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.151+0000 7f08fc4a9700 1 --2- 172.21.15.88:0/3245294107 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f08e00a11e0 0x7f08e00a3690 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f08e4005950 tx=0x7f08e40058e0 comp rx=0 tx=0).stop 2023-07-14T17:43:36.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.151+0000 7f08fc4a9700 1 -- 172.21.15.88:0/3245294107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08f40fe210 msgr2=0x7f08f40ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:36.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.151+0000 7f08fc4a9700 1 --2- 172.21.15.88:0/3245294107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08f40fe210 0x7f08f40ff3f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f08f000c9b0 tx=0x7f08f000ccc0 comp rx=0 tx=0).stop 2023-07-14T17:43:36.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.151+0000 7f08fc4a9700 1 -- 172.21.15.88:0/3245294107 shutdown_connections 2023-07-14T17:43:36.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.151+0000 7f08fc4a9700 1 --2- 172.21.15.88:0/3245294107 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08f4104c70 0x7f08f41a3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:36.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.151+0000 7f08fc4a9700 1 --2- 172.21.15.88:0/3245294107 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f08e00a11e0 0x7f08e00a3690 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:36.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.151+0000 7f08fc4a9700 1 --2- 172.21.15.88:0/3245294107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08f40fe210 0x7f08f40ff3f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:36.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.151+0000 7f08fc4a9700 1 --2- 172.21.15.88:0/3245294107 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08f40febc0 0x7f08f40ff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:36.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.151+0000 7f08fc4a9700 1 -- 172.21.15.88:0/3245294107 >> 172.21.15.88:0/3245294107 conn(0x7f08f4075280 msgr2=0x7f08f40fc960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:36.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.151+0000 7f08fc4a9700 1 -- 172.21.15.88:0/3245294107 shutdown_connections 2023-07-14T17:43:36.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.151+0000 7f08fc4a9700 1 -- 172.21.15.88:0/3245294107 wait complete. 2023-07-14T17:43:36.792 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:36 smithi088 ceph-mon[127465]: pgmap v282: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 11 op/s 2023-07-14T17:43:36.792 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:36 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3245294107' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T17:43:36.792 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:36 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1523482797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T17:43:36.885 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561064 got 188978561066 for osd.6 2023-07-14T17:43:36.886 DEBUG:teuthology.parallel:result is None 2023-07-14T17:43:37.053 INFO:tasks.cephadm.ceph_manager.ceph:need seq 330712481812 got 330712481814 for osd.11 2023-07-14T17:43:37.054 DEBUG:teuthology.parallel:result is None 2023-07-14T17:43:37.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.295+0000 7fb8ee05f700 1 -- 172.21.15.88:0/1798166175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e81008d0 msgr2=0x7fb8e81048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:37.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.295+0000 7fb8ee05f700 1 --2- 172.21.15.88:0/1798166175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e81008d0 0x7fb8e81048a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc0099c0 tx=0x7fb8dc009cd0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.295+0000 7fb8ee05f700 1 -- 172.21.15.88:0/1798166175 shutdown_connections 2023-07-14T17:43:37.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.295+0000 7fb8ee05f700 1 --2- 172.21.15.88:0/1798166175 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8e8104fd0 0x7fb8e810cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.104 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.295+0000 7fb8ee05f700 1 --2- 172.21.15.88:0/1798166175 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb8e80fff30 0x7fb8e8100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.295+0000 7fb8ee05f700 1 --2- 172.21.15.88:0/1798166175 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e81008d0 0x7fb8e81048a0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.295+0000 7fb8ee05f700 1 -- 172.21.15.88:0/1798166175 >> 172.21.15.88:0/1798166175 conn(0x7fb8e80fb790 msgr2=0x7fb8e80fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:37.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.295+0000 7fb8ee05f700 1 -- 172.21.15.88:0/1798166175 shutdown_connections 2023-07-14T17:43:37.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.296+0000 7fb8ee05f700 1 -- 172.21.15.88:0/1798166175 wait complete. 2023-07-14T17:43:37.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.296+0000 7fb8ee05f700 1 Processor -- start 2023-07-14T17:43:37.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.296+0000 7fb8ee05f700 1 -- start start 2023-07-14T17:43:37.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.296+0000 7fb8ee05f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8e80fff30 0x7fb8e8195cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:37.105 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.296+0000 7fb8ee05f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb8e81008d0 0x7fb8e8196200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:37.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.296+0000 7fb8ee05f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e8104fd0 0x7fb8e81995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:37.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.296+0000 7fb8ee05f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e8199bc0 con 0x7fb8e8104fd0 2023-07-14T17:43:37.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.296+0000 7fb8ee05f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e8199d30 con 0x7fb8e80fff30 2023-07-14T17:43:37.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.296+0000 7fb8ee05f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e8199ef0 con 0x7fb8e81008d0 2023-07-14T17:43:37.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.297+0000 7fb8e7fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e8104fd0 0x7fb8e81995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:37.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.297+0000 7fb8e7fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e8104fd0 0x7fb8e81995c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42326/0 (socket says 172.21.15.88:42326) 2023-07-14T17:43:37.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.297+0000 7fb8e7fff700 1 -- 172.21.15.88:0/1523482797 learned_addr learned my addr 172.21.15.88:0/1523482797 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:37.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.297+0000 7fb8e77fe700 1 --2- 172.21.15.88:0/1523482797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8e80fff30 0x7fb8e8195cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:37.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.297+0000 7fb8e6ffd700 1 --2- 172.21.15.88:0/1523482797 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb8e81008d0 0x7fb8e8196200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:37.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.297+0000 7fb8e7fff700 1 -- 172.21.15.88:0/1523482797 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb8e81008d0 msgr2=0x7fb8e8196200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:37.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.297+0000 7fb8e7fff700 1 --2- 172.21.15.88:0/1523482797 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb8e81008d0 0x7fb8e8196200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.297+0000 7fb8e7fff700 1 -- 172.21.15.88:0/1523482797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8e80fff30 msgr2=0x7fb8e8195cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:37.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.297+0000 7fb8e7fff700 1 --2- 172.21.15.88:0/1523482797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8e80fff30 0x7fb8e8195cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.297+0000 7fb8e7fff700 1 -- 172.21.15.88:0/1523482797 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8dc0095f0 con 0x7fb8e8104fd0 2023-07-14T17:43:37.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.297+0000 7fb8e77fe700 1 --2- 172.21.15.88:0/1523482797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8e80fff30 0x7fb8e8195cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:43:37.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.297+0000 7fb8e7fff700 1 --2- 172.21.15.88:0/1523482797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e8104fd0 0x7fb8e81995c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0007ee0 tx=0x7fb8e000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:37.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.297+0000 7fb8e4ff9700 1 -- 172.21.15.88:0/1523482797 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e000ddd0 con 0x7fb8e8104fd0 2023-07-14T17:43:37.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.297+0000 7fb8e4ff9700 1 -- 172.21.15.88:0/1523482797 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb8e000f040 con 0x7fb8e8104fd0 2023-07-14T17:43:37.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.297+0000 7fb8ee05f700 1 -- 172.21.15.88:0/1523482797 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e819a1d0 con 0x7fb8e8104fd0 2023-07-14T17:43:37.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.297+0000 7fb8e4ff9700 1 -- 172.21.15.88:0/1523482797 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e0004870 con 0x7fb8e8104fd0 2023-07-14T17:43:37.108 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.297+0000 7fb8ee05f700 1 -- 172.21.15.88:0/1523482797 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e81a2b20 con 0x7fb8e8104fd0 2023-07-14T17:43:37.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.298+0000 7fb8ee05f700 1 -- 172.21.15.88:0/1523482797 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e804e340 con 0x7fb8e8104fd0 2023-07-14T17:43:37.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.300+0000 7fb8e4ff9700 1 -- 172.21.15.88:0/1523482797 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb8e0003a00 con 0x7fb8e8104fd0 2023-07-14T17:43:37.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.300+0000 7fb8e4ff9700 1 --2- 172.21.15.88:0/1523482797 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb8d00a11e0 0x7fb8d00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:37.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.301+0000 7fb8e4ff9700 1 -- 172.21.15.88:0/1523482797 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb8e00cce30 con 0x7fb8e8104fd0 2023-07-14T17:43:37.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.301+0000 7fb8e77fe700 1 --2- 172.21.15.88:0/1523482797 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb8d00a11e0 0x7fb8d00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:37.109 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.301+0000 7fb8e77fe700 1 --2- 172.21.15.88:0/1523482797 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb8d00a11e0 0x7fb8d00a3690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8005950 tx=0x7fb8d8005f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:37.110 INFO:teuthology.orchestra.run.smithi088.stdout:55834574912 2023-07-14T17:43:37.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.304+0000 7fb8e4ff9700 1 -- 172.21.15.88:0/1523482797 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb8e00961d0 con 0x7fb8e8104fd0 2023-07-14T17:43:37.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.506+0000 7fb8ee05f700 1 -- 172.21.15.88:0/1523482797 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fb8e81a2cb0 con 0x7fb8e8104fd0 2023-07-14T17:43:37.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.506+0000 7fb8e4ff9700 1 -- 172.21.15.88:0/1523482797 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb8e0023090 con 0x7fb8e8104fd0 2023-07-14T17:43:37.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.509+0000 7fb8ee05f700 1 -- 172.21.15.88:0/1523482797 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb8d00a11e0 msgr2=0x7fb8d00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:37.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.509+0000 7fb8ee05f700 1 --2- 172.21.15.88:0/1523482797 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb8d00a11e0 0x7fb8d00a3690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8005950 tx=0x7fb8d8005f30 comp rx=0 tx=0).stop 2023-07-14T17:43:37.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.509+0000 7fb8ee05f700 1 -- 172.21.15.88:0/1523482797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e8104fd0 msgr2=0x7fb8e81995c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:37.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.509+0000 7fb8ee05f700 1 --2- 172.21.15.88:0/1523482797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e8104fd0 0x7fb8e81995c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0007ee0 tx=0x7fb8e000d3b0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.509+0000 7fb8ee05f700 1 -- 172.21.15.88:0/1523482797 shutdown_connections 2023-07-14T17:43:37.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.509+0000 7fb8ee05f700 1 --2- 172.21.15.88:0/1523482797 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb8d00a11e0 0x7fb8d00a3690 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.509+0000 7fb8ee05f700 1 --2- 172.21.15.88:0/1523482797 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8e8104fd0 0x7fb8e81995c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.509+0000 7fb8ee05f700 1 --2- 172.21.15.88:0/1523482797 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb8e80fff30 0x7fb8e8195cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.509+0000 7fb8ee05f700 1 --2- 172.21.15.88:0/1523482797 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb8e81008d0 0x7fb8e8196200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.509+0000 7fb8ee05f700 1 -- 172.21.15.88:0/1523482797 >> 172.21.15.88:0/1523482797 conn(0x7fb8e80fb790 msgr2=0x7fb8e80fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:37.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.509+0000 7fb8ee05f700 1 -- 172.21.15.88:0/1523482797 shutdown_connections 2023-07-14T17:43:37.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:36.509+0000 7fb8ee05f700 1 -- 172.21.15.88:0/1523482797 wait complete. 2023-07-14T17:43:37.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:36 smithi178 ceph-mon[129368]: pgmap v282: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 11 op/s 2023-07-14T17:43:37.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:36 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3245294107' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T17:43:37.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:36 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1523482797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T17:43:37.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:36 smithi107 ceph-mon[128999]: pgmap v282: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 11 op/s 2023-07-14T17:43:37.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:36 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3245294107' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T17:43:37.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:36 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1523482797' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T17:43:37.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.384+0000 7f0c761be700 1 -- 172.21.15.88:0/3874784888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c7010c660 msgr2=0x7f0c7010ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:37.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.384+0000 7f0c761be700 1 --2- 172.21.15.88:0/3874784888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c7010c660 0x7f0c7010ea40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0c6800b0a0 tx=0x7f0c6800b3b0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.384+0000 7f0c761be700 1 -- 172.21.15.88:0/3874784888 shutdown_connections 2023-07-14T17:43:37.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.384+0000 7f0c761be700 1 --2- 172.21.15.88:0/3874784888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c7010c660 0x7f0c7010ea40 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.384+0000 7f0c761be700 1 --2- 172.21.15.88:0/3874784888 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c700feab0 0x7f0c700fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.384+0000 7f0c761be700 1 --2- 172.21.15.88:0/3874784888 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c700ff3c0 0x7f0c7010c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.384+0000 7f0c761be700 1 -- 172.21.15.88:0/3874784888 >> 172.21.15.88:0/3874784888 conn(0x7f0c70075ca0 msgr2=0x7f0c700760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:37.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.384+0000 7f0c761be700 1 -- 172.21.15.88:0/3874784888 shutdown_connections 2023-07-14T17:43:37.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.384+0000 7f0c761be700 1 -- 172.21.15.88:0/3874784888 wait complete. 2023-07-14T17:43:37.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.385+0000 7f0c761be700 1 Processor -- start 2023-07-14T17:43:37.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.385+0000 7f0c761be700 1 -- start start 2023-07-14T17:43:37.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.385+0000 7f0c761be700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c700feab0 0x7f0c70198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:37.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.385+0000 7f0c761be700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c700ff3c0 0x7f0c70198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:37.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.385+0000 7f0c761be700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c7010c660 0x7f0c7019c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:37.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.385+0000 7f0c761be700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7019cfd0 con 0x7f0c700feab0 2023-07-14T17:43:37.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.385+0000 7f0c761be700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7019d140 con 0x7f0c700ff3c0 2023-07-14T17:43:37.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.385+0000 7f0c761be700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7019d300 con 0x7f0c7010c660 2023-07-14T17:43:37.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.386+0000 7f0c6f7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c700feab0 0x7f0c70198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:37.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.386+0000 7f0c6f7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c700feab0 0x7f0c70198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42336/0 (socket says 172.21.15.88:42336) 2023-07-14T17:43:37.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.386+0000 7f0c6f7fe700 1 -- 172.21.15.88:0/3101929510 learned_addr learned my addr 172.21.15.88:0/3101929510 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:37.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.386+0000 7f0c6effd700 1 --2- 172.21.15.88:0/3101929510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c700ff3c0 0x7f0c70198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:37.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.386+0000 7f0c6ffff700 1 --2- 172.21.15.88:0/3101929510 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c7010c660 0x7f0c7019c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:37.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.386+0000 7f0c6f7fe700 1 -- 172.21.15.88:0/3101929510 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c7010c660 msgr2=0x7f0c7019c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:37.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.386+0000 7f0c6f7fe700 1 --2- 172.21.15.88:0/3101929510 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c7010c660 0x7f0c7019c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.386+0000 7f0c6f7fe700 1 -- 172.21.15.88:0/3101929510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c700ff3c0 msgr2=0x7f0c70198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:37.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.386+0000 7f0c6f7fe700 1 --2- 172.21.15.88:0/3101929510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c700ff3c0 0x7f0c70198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.386+0000 7f0c6f7fe700 1 -- 172.21.15.88:0/3101929510 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c68009d00 con 0x7f0c700feab0 2023-07-14T17:43:37.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.386+0000 7f0c6ffff700 1 --2- 172.21.15.88:0/3101929510 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c7010c660 0x7f0c7019c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:43:37.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.386+0000 7f0c6f7fe700 1 --2- 172.21.15.88:0/3101929510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c700feab0 0x7f0c70198010 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0c6000d980 tx=0x7f0c6000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:37.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.386+0000 7f0c6cff9700 1 -- 172.21.15.88:0/3101929510 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c60004220 con 0x7f0c700feab0 2023-07-14T17:43:37.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.386+0000 7f0c6cff9700 1 -- 172.21.15.88:0/3101929510 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0c60007640 con 0x7f0c700feab0 2023-07-14T17:43:37.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.386+0000 7f0c6cff9700 1 -- 172.21.15.88:0/3101929510 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c600055e0 con 0x7f0c700feab0 2023-07-14T17:43:37.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.386+0000 7f0c761be700 1 -- 172.21.15.88:0/3101929510 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c7019d5e0 con 0x7f0c700feab0 2023-07-14T17:43:37.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.386+0000 7f0c761be700 1 -- 172.21.15.88:0/3101929510 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c701a4e40 con 0x7f0c700feab0 2023-07-14T17:43:37.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.387+0000 7f0c761be700 1 -- 172.21.15.88:0/3101929510 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7004e340 con 0x7f0c700feab0 2023-07-14T17:43:37.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.389+0000 7f0c6cff9700 1 -- 172.21.15.88:0/3101929510 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f0c60004750 con 0x7f0c700feab0 2023-07-14T17:43:37.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.390+0000 7f0c6cff9700 1 --2- 172.21.15.88:0/3101929510 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0c580a1260 0x7f0c580a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:37.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.390+0000 7f0c6effd700 1 --2- 172.21.15.88:0/3101929510 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0c580a1260 0x7f0c580a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:37.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.390+0000 7f0c6cff9700 1 -- 172.21.15.88:0/3101929510 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0c600ccf10 con 0x7f0c700feab0 2023-07-14T17:43:37.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.391+0000 7f0c6effd700 1 --2- 172.21.15.88:0/3101929510 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0c580a1260 0x7f0c580a3710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0c70199630 tx=0x7f0c6400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:37.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.393+0000 7f0c6cff9700 1 -- 172.21.15.88:0/3101929510 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0c60096310 con 0x7f0c700feab0 2023-07-14T17:43:37.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.597+0000 7f0c761be700 1 -- 172.21.15.88:0/3101929510 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f0c701a5130 con 0x7f0c700feab0 2023-07-14T17:43:37.580 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.597+0000 7f0c6cff9700 1 -- 172.21.15.88:0/3101929510 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0c60016070 con 0x7f0c700feab0 2023-07-14T17:43:37.580 INFO:teuthology.orchestra.run.smithi088.stdout:274877906974 2023-07-14T17:43:37.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.599+0000 7f0c761be700 1 -- 172.21.15.88:0/3101929510 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0c580a1260 msgr2=0x7f0c580a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:37.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.599+0000 7f0c761be700 1 --2- 172.21.15.88:0/3101929510 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0c580a1260 0x7f0c580a3710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0c70199630 tx=0x7f0c6400a300 comp rx=0 tx=0).stop 2023-07-14T17:43:37.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.599+0000 7f0c761be700 1 -- 172.21.15.88:0/3101929510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c700feab0 msgr2=0x7f0c70198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:37.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.599+0000 7f0c761be700 1 --2- 172.21.15.88:0/3101929510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c700feab0 0x7f0c70198010 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0c6000d980 tx=0x7f0c6000dc90 comp rx=0 tx=0).stop 2023-07-14T17:43:37.582 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.599+0000 7f0c761be700 1 -- 172.21.15.88:0/3101929510 shutdown_connections 2023-07-14T17:43:37.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.599+0000 7f0c761be700 1 --2- 172.21.15.88:0/3101929510 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0c7010c660 0x7f0c7019c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.599+0000 7f0c761be700 1 --2- 172.21.15.88:0/3101929510 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0c580a1260 0x7f0c580a3710 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.599+0000 7f0c761be700 1 --2- 172.21.15.88:0/3101929510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c700feab0 0x7f0c70198010 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.599+0000 7f0c761be700 1 --2- 172.21.15.88:0/3101929510 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0c700ff3c0 0x7f0c70198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:37.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.599+0000 7f0c761be700 1 -- 172.21.15.88:0/3101929510 >> 172.21.15.88:0/3101929510 conn(0x7f0c70075ca0 msgr2=0x7f0c700fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:37.583 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.599+0000 7f0c761be700 1 -- 172.21.15.88:0/3101929510 shutdown_connections 2023-07-14T17:43:37.584 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:37.599+0000 7f0c761be700 1 -- 172.21.15.88:0/3101929510 wait complete. 2023-07-14T17:43:37.941 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:37 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3101929510' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T17:43:38.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:37 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3101929510' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T17:43:38.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:37 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3101929510' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T17:43:38.588 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.457+0000 7f2930d60700 1 -- 172.21.15.88:0/2042310212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c0fef40 msgr2=0x7f292c0ff310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:38.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.457+0000 7f2930d60700 1 --2- 172.21.15.88:0/2042310212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c0fef40 0x7f292c0ff310 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f291c0099c0 tx=0x7f291c009cd0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.457+0000 7f2930d60700 1 -- 172.21.15.88:0/2042310212 shutdown_connections 2023-07-14T17:43:38.589 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.457+0000 7f2930d60700 1 --2- 172.21.15.88:0/2042310212 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f292c101150 0x7f292c10bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.457+0000 7f2930d60700 1 --2- 172.21.15.88:0/2042310212 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c0fef40 0x7f292c0ff310 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.457+0000 7f2930d60700 1 --2- 172.21.15.88:0/2042310212 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f292c1007a0 0x7f292c100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.457+0000 7f2930d60700 1 -- 172.21.15.88:0/2042310212 >> 172.21.15.88:0/2042310212 conn(0x7f292c075c90 msgr2=0x7f292c076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:38.590 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.457+0000 7f2930d60700 1 -- 172.21.15.88:0/2042310212 shutdown_connections 2023-07-14T17:43:38.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.457+0000 7f2930d60700 1 -- 172.21.15.88:0/2042310212 wait complete. 2023-07-14T17:43:38.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.458+0000 7f2930d60700 1 Processor -- start 2023-07-14T17:43:38.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.458+0000 7f2930d60700 1 -- start start 2023-07-14T17:43:38.591 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.458+0000 7f2930d60700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f292c0fef40 0x7f292c196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:38.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.458+0000 7f2930d60700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c1007a0 0x7f292c1971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:38.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.458+0000 7f2930d60700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f292c101150 0x7f292c1a1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:38.592 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.458+0000 7f2930d60700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f292c1a2240 con 0x7f292c1007a0 2023-07-14T17:43:38.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.458+0000 7f2930d60700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f292c1a2380 con 0x7f292c101150 2023-07-14T17:43:38.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.458+0000 7f2930d60700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f292c1a24c0 con 0x7f292c0fef40 2023-07-14T17:43:38.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.458+0000 7f2929d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c1007a0 0x7f292c1971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:38.593 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.458+0000 7f2929d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c1007a0 0x7f292c1971c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42350/0 (socket says 172.21.15.88:42350) 2023-07-14T17:43:38.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.458+0000 7f2929d9b700 1 -- 172.21.15.88:0/3207590350 learned_addr learned my addr 172.21.15.88:0/3207590350 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:38.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.458+0000 7f292a59c700 1 --2- 172.21.15.88:0/3207590350 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f292c0fef40 0x7f292c196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:38.594 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.458+0000 7f292ad9d700 1 --2- 172.21.15.88:0/3207590350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f292c101150 0x7f292c1a1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:38.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.458+0000 7f2929d9b700 1 -- 172.21.15.88:0/3207590350 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f292c0fef40 msgr2=0x7f292c196c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:38.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.458+0000 7f2929d9b700 1 --2- 172.21.15.88:0/3207590350 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f292c0fef40 0x7f292c196c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.458+0000 7f2929d9b700 1 -- 172.21.15.88:0/3207590350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f292c101150 msgr2=0x7f292c1a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:38.595 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.458+0000 7f2929d9b700 1 --2- 172.21.15.88:0/3207590350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f292c101150 0x7f292c1a1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.458+0000 7f2929d9b700 1 -- 172.21.15.88:0/3207590350 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291c0095f0 con 0x7f292c1007a0 2023-07-14T17:43:38.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.459+0000 7f2929d9b700 1 --2- 172.21.15.88:0/3207590350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c1007a0 0x7f292c1971c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f292000d980 tx=0x7f292000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:38.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.459+0000 7f291b7fe700 1 -- 172.21.15.88:0/3207590350 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2920009b40 con 0x7f292c1007a0 2023-07-14T17:43:38.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.459+0000 7f2930d60700 1 -- 172.21.15.88:0/3207590350 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f292c1a2720 con 0x7f292c1007a0 2023-07-14T17:43:38.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.459+0000 7f291b7fe700 1 -- 172.21.15.88:0/3207590350 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f292000b6e0 con 0x7f292c1007a0 2023-07-14T17:43:38.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.459+0000 7f291b7fe700 1 -- 172.21.15.88:0/3207590350 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292000f3c0 con 0x7f292c1007a0 2023-07-14T17:43:38.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.459+0000 7f2930d60700 1 -- 172.21.15.88:0/3207590350 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f292c1a2c70 con 0x7f292c1007a0 2023-07-14T17:43:38.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.460+0000 7f2930d60700 1 -- 172.21.15.88:0/3207590350 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f292c04e340 con 0x7f292c1007a0 2023-07-14T17:43:38.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.461+0000 7f291b7fe700 1 -- 172.21.15.88:0/3207590350 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2920010460 con 0x7f292c1007a0 2023-07-14T17:43:38.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.461+0000 7f291b7fe700 1 --2- 172.21.15.88:0/3207590350 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f29140a11e0 0x7f29140a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:38.598 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.461+0000 7f292a59c700 1 --2- 172.21.15.88:0/3207590350 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f29140a11e0 0x7f29140a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:38.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.461+0000 7f291b7fe700 1 -- 172.21.15.88:0/3207590350 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f29200ccb40 con 0x7f292c1007a0 2023-07-14T17:43:38.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.462+0000 7f292a59c700 1 --2- 172.21.15.88:0/3207590350 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f29140a11e0 0x7f29140a3690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f291c006010 tx=0x7f291c003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:38.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.465+0000 7f291b7fe700 1 -- 172.21.15.88:0/3207590350 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2920095f40 con 0x7f292c1007a0 2023-07-14T17:43:38.609 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574911 got 55834574912 for osd.1 2023-07-14T17:43:38.610 DEBUG:teuthology.parallel:result is None 2023-07-14T17:43:38.616 INFO:tasks.cephadm.ceph_manager.ceph:need seq 274877906972 got 274877906974 for osd.9 2023-07-14T17:43:38.616 DEBUG:teuthology.parallel:result is None 2023-07-14T17:43:38.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.685+0000 7f2930d60700 1 -- 172.21.15.88:0/3207590350 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f292c1a2e90 con 0x7f292c1007a0 2023-07-14T17:43:38.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.685+0000 7f291b7fe700 1 -- 172.21.15.88:0/3207590350 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2920016090 con 0x7f292c1007a0 2023-07-14T17:43:38.669 INFO:teuthology.orchestra.run.smithi088.stdout:158913790000 2023-07-14T17:43:38.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.687+0000 7f2930d60700 1 -- 172.21.15.88:0/3207590350 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f29140a11e0 msgr2=0x7f29140a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:38.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.687+0000 7f2930d60700 1 --2- 172.21.15.88:0/3207590350 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f29140a11e0 0x7f29140a3690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f291c006010 tx=0x7f291c003680 comp rx=0 tx=0).stop 2023-07-14T17:43:38.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.687+0000 7f2930d60700 1 -- 172.21.15.88:0/3207590350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c1007a0 msgr2=0x7f292c1971c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:38.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.687+0000 7f2930d60700 1 --2- 172.21.15.88:0/3207590350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c1007a0 0x7f292c1971c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f292000d980 tx=0x7f292000dc90 comp rx=0 tx=0).stop 2023-07-14T17:43:38.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.687+0000 7f2930d60700 1 -- 172.21.15.88:0/3207590350 shutdown_connections 2023-07-14T17:43:38.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.687+0000 7f2930d60700 1 --2- 172.21.15.88:0/3207590350 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f292c101150 0x7f292c1a1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.687+0000 7f2930d60700 1 --2- 172.21.15.88:0/3207590350 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f292c0fef40 0x7f292c196c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.687+0000 7f2930d60700 1 --2- 172.21.15.88:0/3207590350 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f29140a11e0 0x7f29140a3690 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.687+0000 7f2930d60700 1 --2- 172.21.15.88:0/3207590350 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f292c1007a0 0x7f292c1971c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.687+0000 7f2930d60700 1 -- 172.21.15.88:0/3207590350 >> 172.21.15.88:0/3207590350 conn(0x7f292c075c90 msgr2=0x7f292c0fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:38.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.688+0000 7f2930d60700 1 -- 172.21.15.88:0/3207590350 shutdown_connections 2023-07-14T17:43:38.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.688+0000 7f2930d60700 1 -- 172.21.15.88:0/3207590350 wait complete. 2023-07-14T17:43:38.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.767+0000 7f7b6acfb700 1 -- 172.21.15.88:0/3986837049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b64101830 msgr2=0x7f7b64109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:38.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.767+0000 7f7b6acfb700 1 --2- 172.21.15.88:0/3986837049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b64101830 0x7f7b64109d30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7b50009990 tx=0x7f7b50009ca0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.767+0000 7f7b6acfb700 1 -- 172.21.15.88:0/3986837049 shutdown_connections 2023-07-14T17:43:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.767+0000 7f7b6acfb700 1 --2- 172.21.15.88:0/3986837049 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b6410a270 0x7f7b6410c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.767+0000 7f7b6acfb700 1 --2- 172.21.15.88:0/3986837049 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b64100f20 0x7f7b641012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.767+0000 7f7b6acfb700 1 --2- 172.21.15.88:0/3986837049 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b64101830 0x7f7b64109d30 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.768+0000 7f7b6acfb700 1 -- 172.21.15.88:0/3986837049 >> 172.21.15.88:0/3986837049 conn(0x7f7b64075ca0 msgr2=0x7f7b640760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.768+0000 7f7b6acfb700 1 -- 172.21.15.88:0/3986837049 shutdown_connections 2023-07-14T17:43:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.768+0000 7f7b6acfb700 1 -- 172.21.15.88:0/3986837049 wait complete. 2023-07-14T17:43:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.768+0000 7f7b6acfb700 1 Processor -- start 2023-07-14T17:43:38.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.768+0000 7f7b6acfb700 1 -- start start 2023-07-14T17:43:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.768+0000 7f7b6acfb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b64100f20 0x7f7b64195cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.768+0000 7f7b6acfb700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b64101830 0x7f7b641961f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.768+0000 7f7b6acfb700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b6410a270 0x7f7b641995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.768+0000 7f7b6acfb700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b64199bb0 con 0x7f7b64101830 2023-07-14T17:43:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.768+0000 7f7b6acfb700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b64199d20 con 0x7f7b64100f20 2023-07-14T17:43:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.768+0000 7f7b6acfb700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b64199ee0 con 0x7f7b6410a270 2023-07-14T17:43:38.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.768+0000 7f7b63fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b64101830 0x7f7b641961f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:38.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.769+0000 7f7b63fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b64101830 0x7f7b641961f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42364/0 (socket says 172.21.15.88:42364) 2023-07-14T17:43:38.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.769+0000 7f7b63fff700 1 -- 172.21.15.88:0/3992290911 learned_addr learned my addr 172.21.15.88:0/3992290911 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:38.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.769+0000 7f7b68a97700 1 --2- 172.21.15.88:0/3992290911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b64100f20 0x7f7b64195cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:38.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.769+0000 7f7b69298700 1 --2- 172.21.15.88:0/3992290911 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b6410a270 0x7f7b641995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:38.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.769+0000 7f7b63fff700 1 -- 172.21.15.88:0/3992290911 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b6410a270 msgr2=0x7f7b641995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:38.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.769+0000 7f7b63fff700 1 --2- 172.21.15.88:0/3992290911 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b6410a270 0x7f7b641995b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.769+0000 7f7b63fff700 1 -- 172.21.15.88:0/3992290911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b64100f20 msgr2=0x7f7b64195cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:38.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.769+0000 7f7b63fff700 1 --2- 172.21.15.88:0/3992290911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b64100f20 0x7f7b64195cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.769+0000 7f7b63fff700 1 -- 172.21.15.88:0/3992290911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b500095f0 con 0x7f7b64101830 2023-07-14T17:43:38.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.769+0000 7f7b63fff700 1 --2- 172.21.15.88:0/3992290911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b64101830 0x7f7b641961f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7b50000c00 tx=0x7f7b50004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:38.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.769+0000 7f7b61ffb700 1 -- 172.21.15.88:0/3992290911 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b5001d070 con 0x7f7b64101830 2023-07-14T17:43:38.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.769+0000 7f7b6acfb700 1 -- 172.21.15.88:0/3992290911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6419a160 con 0x7f7b64101830 2023-07-14T17:43:38.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.769+0000 7f7b61ffb700 1 -- 172.21.15.88:0/3992290911 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7b50022470 con 0x7f7b64101830 2023-07-14T17:43:38.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.769+0000 7f7b61ffb700 1 -- 172.21.15.88:0/3992290911 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b50017650 con 0x7f7b64101830 2023-07-14T17:43:38.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.769+0000 7f7b6acfb700 1 -- 172.21.15.88:0/3992290911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b641a2b10 con 0x7f7b64101830 2023-07-14T17:43:38.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.771+0000 7f7b6acfb700 1 -- 172.21.15.88:0/3992290911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b64104b00 con 0x7f7b64101830 2023-07-14T17:43:38.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.773+0000 7f7b61ffb700 1 -- 172.21.15.88:0/3992290911 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7b500225e0 con 0x7f7b64101830 2023-07-14T17:43:38.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.774+0000 7f7b61ffb700 1 --2- 172.21.15.88:0/3992290911 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7b540a11e0 0x7f7b540a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:38.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.774+0000 7f7b68a97700 1 --2- 172.21.15.88:0/3992290911 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7b540a11e0 0x7f7b540a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:38.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.774+0000 7f7b61ffb700 1 -- 172.21.15.88:0/3992290911 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7b500ce880 con 0x7f7b64101830 2023-07-14T17:43:38.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.774+0000 7f7b68a97700 1 --2- 172.21.15.88:0/3992290911 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7b540a11e0 0x7f7b540a3690 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7b58009c60 tx=0x7f7b58009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:38.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.777+0000 7f7b61ffb700 1 -- 172.21.15.88:0/3992290911 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7b50097d10 con 0x7f7b64101830 2023-07-14T17:43:38.941 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:38 smithi088 ceph-mon[127465]: pgmap v283: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2023-07-14T17:43:38.942 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:38 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3207590350' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T17:43:38.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.989+0000 7f7b6acfb700 1 -- 172.21.15.88:0/3992290911 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f7b641044e0 con 0x7f7b64101830 2023-07-14T17:43:38.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.990+0000 7f7b61ffb700 1 -- 172.21.15.88:0/3992290911 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7b5009b330 con 0x7f7b64101830 2023-07-14T17:43:38.973 INFO:teuthology.orchestra.run.smithi088.stdout:107374182457 2023-07-14T17:43:38.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.992+0000 7f7b6acfb700 1 -- 172.21.15.88:0/3992290911 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7b540a11e0 msgr2=0x7f7b540a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:38.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.992+0000 7f7b6acfb700 1 --2- 172.21.15.88:0/3992290911 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7b540a11e0 0x7f7b540a3690 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7b58009c60 tx=0x7f7b58009260 comp rx=0 tx=0).stop 2023-07-14T17:43:38.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.992+0000 7f7b6acfb700 1 -- 172.21.15.88:0/3992290911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b64101830 msgr2=0x7f7b641961f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:38.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.992+0000 7f7b6acfb700 1 --2- 172.21.15.88:0/3992290911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b64101830 0x7f7b641961f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7b50000c00 tx=0x7f7b50004730 comp rx=0 tx=0).stop 2023-07-14T17:43:38.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.992+0000 7f7b6acfb700 1 -- 172.21.15.88:0/3992290911 shutdown_connections 2023-07-14T17:43:38.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.992+0000 7f7b6acfb700 1 --2- 172.21.15.88:0/3992290911 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7b6410a270 0x7f7b641995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.992+0000 7f7b6acfb700 1 --2- 172.21.15.88:0/3992290911 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7b64100f20 0x7f7b64195cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.992+0000 7f7b6acfb700 1 --2- 172.21.15.88:0/3992290911 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7b540a11e0 0x7f7b540a3690 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.992+0000 7f7b6acfb700 1 --2- 172.21.15.88:0/3992290911 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b64101830 0x7f7b641961f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.992+0000 7f7b6acfb700 1 -- 172.21.15.88:0/3992290911 >> 172.21.15.88:0/3992290911 conn(0x7f7b64075ca0 msgr2=0x7f7b64101c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:38.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.992+0000 7f7b6acfb700 1 -- 172.21.15.88:0/3992290911 shutdown_connections 2023-07-14T17:43:38.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:38.992+0000 7f7b6acfb700 1 -- 172.21.15.88:0/3992290911 wait complete. 2023-07-14T17:43:38.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.003+0000 7feff9cfe700 1 -- 172.21.15.88:0/923046896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feff40feec0 msgr2=0x7feff40ff290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:38.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.003+0000 7feff9cfe700 1 --2- 172.21.15.88:0/923046896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feff40feec0 0x7feff40ff290 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fefe40099c0 tx=0x7fefe4009cd0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.004+0000 7feff9cfe700 1 -- 172.21.15.88:0/923046896 shutdown_connections 2023-07-14T17:43:38.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.004+0000 7feff9cfe700 1 --2- 172.21.15.88:0/923046896 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff4103f60 0x7feff410bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.004+0000 7feff9cfe700 1 --2- 172.21.15.88:0/923046896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feff40feec0 0x7feff40ff290 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.004+0000 7feff9cfe700 1 --2- 172.21.15.88:0/923046896 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feff40ff860 0x7feff4103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.004+0000 7feff9cfe700 1 -- 172.21.15.88:0/923046896 >> 172.21.15.88:0/923046896 conn(0x7feff40fa720 msgr2=0x7feff40fcb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:38.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.004+0000 7feff9cfe700 1 -- 172.21.15.88:0/923046896 shutdown_connections 2023-07-14T17:43:38.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.004+0000 7feff9cfe700 1 -- 172.21.15.88:0/923046896 wait complete. 2023-07-14T17:43:38.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.005+0000 7feff9cfe700 1 Processor -- start 2023-07-14T17:43:38.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.005+0000 7feff9cfe700 1 -- start start 2023-07-14T17:43:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.005+0000 7feff9cfe700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff40feec0 0x7feff419a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.005+0000 7feff9cfe700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feff40ff860 0x7feff419af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.005+0000 7feff9cfe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feff4103f60 0x7feff4194a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.005+0000 7feff9cfe700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff41950c0 con 0x7feff4103f60 2023-07-14T17:43:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.005+0000 7feff9cfe700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff4195230 con 0x7feff40feec0 2023-07-14T17:43:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.005+0000 7feff9cfe700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff41953f0 con 0x7feff40ff860 2023-07-14T17:43:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.005+0000 7feff3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feff4103f60 0x7feff4194a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:38.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.005+0000 7feff3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feff4103f60 0x7feff4194a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42380/0 (socket says 172.21.15.88:42380) 2023-07-14T17:43:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.005+0000 7feff3fff700 1 -- 172.21.15.88:0/4217589333 learned_addr learned my addr 172.21.15.88:0/4217589333 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.005+0000 7feff37fe700 1 --2- 172.21.15.88:0/4217589333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff40feec0 0x7feff419a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.006+0000 7feff2ffd700 1 --2- 172.21.15.88:0/4217589333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feff40ff860 0x7feff419af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.006+0000 7feff3fff700 1 -- 172.21.15.88:0/4217589333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feff40ff860 msgr2=0x7feff419af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.006+0000 7feff3fff700 1 --2- 172.21.15.88:0/4217589333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feff40ff860 0x7feff419af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.006+0000 7feff3fff700 1 -- 172.21.15.88:0/4217589333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff40feec0 msgr2=0x7feff419a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:38.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.006+0000 7feff3fff700 1 --2- 172.21.15.88:0/4217589333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff40feec0 0x7feff419a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.006+0000 7feff3fff700 1 -- 172.21.15.88:0/4217589333 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe40095f0 con 0x7feff4103f60 2023-07-14T17:43:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.006+0000 7feff37fe700 1 --2- 172.21.15.88:0/4217589333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff40feec0 0x7feff419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T17:43:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.006+0000 7feff3fff700 1 --2- 172.21.15.88:0/4217589333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feff4103f60 0x7feff4194a50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fefec00ef30 tx=0x7fefec00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.006+0000 7feff0ff9700 1 -- 172.21.15.88:0/4217589333 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec018070 con 0x7feff4103f60 2023-07-14T17:43:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.006+0000 7feff0ff9700 1 -- 172.21.15.88:0/4217589333 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fefec00f040 con 0x7feff4103f60 2023-07-14T17:43:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.006+0000 7feff9cfe700 1 -- 172.21.15.88:0/4217589333 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff4195670 con 0x7feff4103f60 2023-07-14T17:43:38.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.006+0000 7feff0ff9700 1 -- 172.21.15.88:0/4217589333 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec00cc70 con 0x7feff4103f60 2023-07-14T17:43:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.006+0000 7feff9cfe700 1 -- 172.21.15.88:0/4217589333 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff41a19c0 con 0x7feff4103f60 2023-07-14T17:43:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.007+0000 7feff9cfe700 1 -- 172.21.15.88:0/4217589333 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff4107140 con 0x7feff4103f60 2023-07-14T17:43:38.993 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.009+0000 7feff0ff9700 1 -- 172.21.15.88:0/4217589333 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fefec003a00 con 0x7feff4103f60 2023-07-14T17:43:38.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.009+0000 7feff0ff9700 1 --2- 172.21.15.88:0/4217589333 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fefdc0a11e0 0x7fefdc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:38.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.009+0000 7feff37fe700 1 --2- 172.21.15.88:0/4217589333 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fefdc0a11e0 0x7fefdc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:38.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.009+0000 7feff0ff9700 1 -- 172.21.15.88:0/4217589333 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fefec0ccd90 con 0x7feff4103f60 2023-07-14T17:43:38.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.010+0000 7feff37fe700 1 --2- 172.21.15.88:0/4217589333 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fefdc0a11e0 0x7fefdc0a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fefe400b580 tx=0x7fefe401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:38.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.014+0000 7feff0ff9700 1 -- 172.21.15.88:0/4217589333 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fefec096220 con 0x7feff4103f60 2023-07-14T17:43:39.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.099+0000 7f8634af9700 1 -- 172.21.15.88:0/2171976954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86300ff5a0 msgr2=0x7f86300ff970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:39.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.099+0000 7f8634af9700 1 --2- 172.21.15.88:0/2171976954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86300ff5a0 0x7f86300ff970 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f86200099c0 tx=0x7f8620009cd0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.099+0000 7f8634af9700 1 -- 172.21.15.88:0/2171976954 shutdown_connections 2023-07-14T17:43:39.082 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.099+0000 7f8634af9700 1 --2- 172.21.15.88:0/2171976954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8630104120 0x7f863010bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.099+0000 7f8634af9700 1 --2- 172.21.15.88:0/2171976954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86300ff5a0 0x7f86300ff970 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.099+0000 7f8634af9700 1 --2- 172.21.15.88:0/2171976954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f86300fff40 0x7f86301039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.099+0000 7f8634af9700 1 -- 172.21.15.88:0/2171976954 >> 172.21.15.88:0/2171976954 conn(0x7f8630075280 msgr2=0x7f8630075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:39.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.099+0000 7f8634af9700 1 -- 172.21.15.88:0/2171976954 shutdown_connections 2023-07-14T17:43:39.083 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.099+0000 7f8634af9700 1 -- 172.21.15.88:0/2171976954 wait complete. 2023-07-14T17:43:39.084 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.100+0000 7f8634af9700 1 Processor -- start 2023-07-14T17:43:39.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.100+0000 7f8634af9700 1 -- start start 2023-07-14T17:43:39.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.100+0000 7f8634af9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86300ff5a0 0x7f8630194dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:39.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.100+0000 7f8634af9700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86300fff40 0x7f8630195310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:39.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.100+0000 7f8634af9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8630104120 0x7f8630199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:39.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.100+0000 7f8634af9700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8630199d90 con 0x7f86300ff5a0 2023-07-14T17:43:39.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.100+0000 7f8634af9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8630199f00 con 0x7f8630104120 2023-07-14T17:43:39.085 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.100+0000 7f8634af9700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863019a0c0 con 0x7f86300fff40 2023-07-14T17:43:39.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.100+0000 7f862e59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86300ff5a0 0x7f8630194dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:39.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.100+0000 7f862e59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86300ff5a0 0x7f8630194dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42390/0 (socket says 172.21.15.88:42390) 2023-07-14T17:43:39.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.100+0000 7f862e59c700 1 -- 172.21.15.88:0/2434278703 learned_addr learned my addr 172.21.15.88:0/2434278703 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:39.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.100+0000 7f862ed9d700 1 --2- 172.21.15.88:0/2434278703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8630104120 0x7f8630199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:39.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.100+0000 7f862dd9b700 1 --2- 172.21.15.88:0/2434278703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86300fff40 0x7f8630195310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:39.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.101+0000 7f862e59c700 1 -- 172.21.15.88:0/2434278703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86300fff40 msgr2=0x7f8630195310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:39.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.101+0000 7f862e59c700 1 --2- 172.21.15.88:0/2434278703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86300fff40 0x7f8630195310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.086 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.101+0000 7f862e59c700 1 -- 172.21.15.88:0/2434278703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8630104120 msgr2=0x7f8630199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:39.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.101+0000 7f862e59c700 1 --2- 172.21.15.88:0/2434278703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8630104120 0x7f8630199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.101+0000 7f862e59c700 1 -- 172.21.15.88:0/2434278703 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86200095f0 con 0x7f86300ff5a0 2023-07-14T17:43:39.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.101+0000 7f862dd9b700 1 --2- 172.21.15.88:0/2434278703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86300fff40 0x7f8630195310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:43:39.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.101+0000 7f862e59c700 1 --2- 172.21.15.88:0/2434278703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86300ff5a0 0x7f8630194dd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8620005910 tx=0x7f8620009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:39.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.101+0000 7f861f7fe700 1 -- 172.21.15.88:0/2434278703 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862001d070 con 0x7f86300ff5a0 2023-07-14T17:43:39.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.101+0000 7f8634af9700 1 -- 172.21.15.88:0/2434278703 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f863019a340 con 0x7f86300ff5a0 2023-07-14T17:43:39.087 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.101+0000 7f861f7fe700 1 -- 172.21.15.88:0/2434278703 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f86200216e0 con 0x7f86300ff5a0 2023-07-14T17:43:39.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.101+0000 7f861f7fe700 1 -- 172.21.15.88:0/2434278703 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8620017650 con 0x7f86300ff5a0 2023-07-14T17:43:39.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.101+0000 7f8634af9700 1 -- 172.21.15.88:0/2434278703 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86300fe400 con 0x7f86300ff5a0 2023-07-14T17:43:39.088 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.102+0000 7f8634af9700 1 -- 172.21.15.88:0/2434278703 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f863004e340 con 0x7f86300ff5a0 2023-07-14T17:43:39.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.104+0000 7f861f7fe700 1 -- 172.21.15.88:0/2434278703 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8620021850 con 0x7f86300ff5a0 2023-07-14T17:43:39.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.104+0000 7f861f7fe700 1 --2- 172.21.15.88:0/2434278703 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f86180a11e0 0x7f86180a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:39.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.105+0000 7f861f7fe700 1 -- 172.21.15.88:0/2434278703 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f86200d1c70 con 0x7f86300ff5a0 2023-07-14T17:43:39.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.105+0000 7f862dd9b700 1 --2- 172.21.15.88:0/2434278703 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f86180a11e0 0x7f86180a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:39.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.105+0000 7f862dd9b700 1 --2- 172.21.15.88:0/2434278703 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f86180a11e0 0x7f86180a3690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f86301963f0 tx=0x7f8624009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:39.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.113+0000 7f861f7fe700 1 -- 172.21.15.88:0/2434278703 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f862009b100 con 0x7f86300ff5a0 2023-07-14T17:43:39.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:38 smithi178 ceph-mon[129368]: pgmap v283: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2023-07-14T17:43:39.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:38 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3207590350' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T17:43:39.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.185+0000 7fc595dd9700 1 -- 172.21.15.88:0/2604063164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5900feca0 msgr2=0x7fc59010e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:39.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.185+0000 7fc595dd9700 1 --2- 172.21.15.88:0/2604063164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5900feca0 0x7fc59010e9a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc58800b0a0 tx=0x7fc58800b3b0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.185+0000 7fc595dd9700 1 -- 172.21.15.88:0/2604063164 shutdown_connections 2023-07-14T17:43:39.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.185+0000 7fc595dd9700 1 --2- 172.21.15.88:0/2604063164 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5900feca0 0x7fc59010e9a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.185+0000 7fc595dd9700 1 --2- 172.21.15.88:0/2604063164 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc590100b90 0x7fc590100f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.185+0000 7fc595dd9700 1 --2- 172.21.15.88:0/2604063164 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5900fe210 0x7fc5900fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.185+0000 7fc595dd9700 1 -- 172.21.15.88:0/2604063164 >> 172.21.15.88:0/2604063164 conn(0x7fc590075280 msgr2=0x7fc590075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:39.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.185+0000 7fc595dd9700 1 -- 172.21.15.88:0/2604063164 shutdown_connections 2023-07-14T17:43:39.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.185+0000 7fc595dd9700 1 -- 172.21.15.88:0/2604063164 wait complete. 2023-07-14T17:43:39.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.185+0000 7fc595dd9700 1 Processor -- start 2023-07-14T17:43:39.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc595dd9700 1 -- start start 2023-07-14T17:43:39.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc595dd9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5900fe210 0x7fc590197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:39.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc595dd9700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5900feca0 0x7fc5901984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:39.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc595dd9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc590100b90 0x7fc59019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:39.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc595dd9700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59019cf70 con 0x7fc590100b90 2023-07-14T17:43:39.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc595dd9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59019d0e0 con 0x7fc5900fe210 2023-07-14T17:43:39.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc595dd9700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59019d2a0 con 0x7fc5900feca0 2023-07-14T17:43:39.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc58ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc590100b90 0x7fc59019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:39.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc58ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc590100b90 0x7fc59019c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42406/0 (socket says 172.21.15.88:42406) 2023-07-14T17:43:39.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc58ffff700 1 -- 172.21.15.88:0/3782149062 learned_addr learned my addr 172.21.15.88:0/3782149062 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:39.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc58effd700 1 --2- 172.21.15.88:0/3782149062 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5900feca0 0x7fc5901984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:39.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc58f7fe700 1 --2- 172.21.15.88:0/3782149062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5900fe210 0x7fc590197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:39.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc58ffff700 1 -- 172.21.15.88:0/3782149062 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5900feca0 msgr2=0x7fc5901984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:39.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc58ffff700 1 --2- 172.21.15.88:0/3782149062 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5900feca0 0x7fc5901984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc58ffff700 1 -- 172.21.15.88:0/3782149062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5900fe210 msgr2=0x7fc590197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:39.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc58ffff700 1 --2- 172.21.15.88:0/3782149062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5900fe210 0x7fc590197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc58ffff700 1 -- 172.21.15.88:0/3782149062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc588009d00 con 0x7fc590100b90 2023-07-14T17:43:39.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc58f7fe700 1 --2- 172.21.15.88:0/3782149062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5900fe210 0x7fc590197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:43:39.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.186+0000 7fc58ffff700 1 --2- 172.21.15.88:0/3782149062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc590100b90 0x7fc59019c960 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc58800b780 tx=0x7fc58800bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:39.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.187+0000 7fc58cff9700 1 -- 172.21.15.88:0/3782149062 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58800e040 con 0x7fc590100b90 2023-07-14T17:43:39.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.187+0000 7fc58cff9700 1 -- 172.21.15.88:0/3782149062 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc588004e60 con 0x7fc590100b90 2023-07-14T17:43:39.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.187+0000 7fc595dd9700 1 -- 172.21.15.88:0/3782149062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59019d520 con 0x7fc590100b90 2023-07-14T17:43:39.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.187+0000 7fc58cff9700 1 -- 172.21.15.88:0/3782149062 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5880052a0 con 0x7fc590100b90 2023-07-14T17:43:39.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.187+0000 7fc595dd9700 1 -- 172.21.15.88:0/3782149062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5901a4de0 con 0x7fc590100b90 2023-07-14T17:43:39.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.188+0000 7fc595dd9700 1 -- 172.21.15.88:0/3782149062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc59004e340 con 0x7fc590100b90 2023-07-14T17:43:39.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.190+0000 7fc58cff9700 1 -- 172.21.15.88:0/3782149062 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc588005400 con 0x7fc590100b90 2023-07-14T17:43:39.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.190+0000 7fc58cff9700 1 --2- 172.21.15.88:0/3782149062 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc5780a1260 0x7fc5780a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:39.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.190+0000 7fc58f7fe700 1 --2- 172.21.15.88:0/3782149062 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc5780a1260 0x7fc5780a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:39.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.191+0000 7fc58cff9700 1 -- 172.21.15.88:0/3782149062 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc5880ce7b0 con 0x7fc590100b90 2023-07-14T17:43:39.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.191+0000 7fc58f7fe700 1 --2- 172.21.15.88:0/3782149062 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc5780a1260 0x7fc5780a3710 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc58000bb90 tx=0x7fc580005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:39.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.194+0000 7fc58cff9700 1 -- 172.21.15.88:0/3782149062 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc588097c40 con 0x7fc590100b90 2023-07-14T17:43:39.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.225+0000 7feff9cfe700 1 -- 172.21.15.88:0/4217589333 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7feff40682d0 con 0x7feff4103f60 2023-07-14T17:43:39.208 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.225+0000 7feff0ff9700 1 -- 172.21.15.88:0/4217589333 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fefec099840 con 0x7feff4103f60 2023-07-14T17:43:39.209 INFO:teuthology.orchestra.run.smithi088.stdout:244813135907 2023-07-14T17:43:39.210 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.228+0000 7feff9cfe700 1 -- 172.21.15.88:0/4217589333 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fefdc0a11e0 msgr2=0x7fefdc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:39.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.228+0000 7feff9cfe700 1 --2- 172.21.15.88:0/4217589333 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fefdc0a11e0 0x7fefdc0a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fefe400b580 tx=0x7fefe401a040 comp rx=0 tx=0).stop 2023-07-14T17:43:39.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.228+0000 7feff9cfe700 1 -- 172.21.15.88:0/4217589333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feff4103f60 msgr2=0x7feff4194a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:39.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.228+0000 7feff9cfe700 1 --2- 172.21.15.88:0/4217589333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feff4103f60 0x7feff4194a50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fefec00ef30 tx=0x7fefec00c3a0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.228+0000 7feff9cfe700 1 -- 172.21.15.88:0/4217589333 shutdown_connections 2023-07-14T17:43:39.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.228+0000 7feff9cfe700 1 --2- 172.21.15.88:0/4217589333 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fefdc0a11e0 0x7fefdc0a3690 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.228+0000 7feff9cfe700 1 --2- 172.21.15.88:0/4217589333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feff4103f60 0x7feff4194a50 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.211 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.228+0000 7feff9cfe700 1 --2- 172.21.15.88:0/4217589333 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feff40feec0 0x7feff419a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.228+0000 7feff9cfe700 1 --2- 172.21.15.88:0/4217589333 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feff40ff860 0x7feff419af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.228+0000 7feff9cfe700 1 -- 172.21.15.88:0/4217589333 >> 172.21.15.88:0/4217589333 conn(0x7feff40fa720 msgr2=0x7feff40fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:39.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.228+0000 7feff9cfe700 1 -- 172.21.15.88:0/4217589333 shutdown_connections 2023-07-14T17:43:39.212 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.228+0000 7feff9cfe700 1 -- 172.21.15.88:0/4217589333 wait complete. 2023-07-14T17:43:39.223 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:38 smithi107 ceph-mon[128999]: pgmap v283: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2023-07-14T17:43:39.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:38 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3207590350' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T17:43:39.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.310+0000 7f8634af9700 1 -- 172.21.15.88:0/2434278703 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f8630067ee0 con 0x7f86300ff5a0 2023-07-14T17:43:39.293 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.310+0000 7f861f7fe700 1 -- 172.21.15.88:0/2434278703 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f862009e720 con 0x7f86300ff5a0 2023-07-14T17:43:39.294 INFO:teuthology.orchestra.run.smithi088.stdout:296352743450 2023-07-14T17:43:39.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.313+0000 7f8634af9700 1 -- 172.21.15.88:0/2434278703 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f86180a11e0 msgr2=0x7f86180a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:39.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.313+0000 7f8634af9700 1 --2- 172.21.15.88:0/2434278703 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f86180a11e0 0x7f86180a3690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f86301963f0 tx=0x7f8624009260 comp rx=0 tx=0).stop 2023-07-14T17:43:39.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.313+0000 7f8634af9700 1 -- 172.21.15.88:0/2434278703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86300ff5a0 msgr2=0x7f8630194dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:39.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.313+0000 7f8634af9700 1 --2- 172.21.15.88:0/2434278703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86300ff5a0 0x7f8630194dd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8620005910 tx=0x7f8620009580 comp rx=0 tx=0).stop 2023-07-14T17:43:39.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.313+0000 7f8634af9700 1 -- 172.21.15.88:0/2434278703 shutdown_connections 2023-07-14T17:43:39.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.314+0000 7f8634af9700 1 --2- 172.21.15.88:0/2434278703 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8630104120 0x7f8630199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.314+0000 7f8634af9700 1 --2- 172.21.15.88:0/2434278703 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f86180a11e0 0x7f86180a3690 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.314+0000 7f8634af9700 1 --2- 172.21.15.88:0/2434278703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86300ff5a0 0x7f8630194dd0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.314+0000 7f8634af9700 1 --2- 172.21.15.88:0/2434278703 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f86300fff40 0x7f8630195310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.314+0000 7f8634af9700 1 -- 172.21.15.88:0/2434278703 >> 172.21.15.88:0/2434278703 conn(0x7f8630075280 msgr2=0x7f86300fbe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:39.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.314+0000 7f8634af9700 1 -- 172.21.15.88:0/2434278703 shutdown_connections 2023-07-14T17:43:39.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.314+0000 7f8634af9700 1 -- 172.21.15.88:0/2434278703 wait complete. 2023-07-14T17:43:39.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.402+0000 7fc595dd9700 1 -- 172.21.15.88:0/3782149062 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fc5901a4f70 con 0x7fc590100b90 2023-07-14T17:43:39.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.402+0000 7fc58cff9700 1 -- 172.21.15.88:0/3782149062 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc58809b260 con 0x7fc590100b90 2023-07-14T17:43:39.385 INFO:teuthology.orchestra.run.smithi088.stdout:77309411388 2023-07-14T17:43:39.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.404+0000 7fc595dd9700 1 -- 172.21.15.88:0/3782149062 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc5780a1260 msgr2=0x7fc5780a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:39.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.404+0000 7fc595dd9700 1 --2- 172.21.15.88:0/3782149062 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc5780a1260 0x7fc5780a3710 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc58000bb90 tx=0x7fc580005d80 comp rx=0 tx=0).stop 2023-07-14T17:43:39.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.404+0000 7fc595dd9700 1 -- 172.21.15.88:0/3782149062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc590100b90 msgr2=0x7fc59019c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:39.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.404+0000 7fc595dd9700 1 --2- 172.21.15.88:0/3782149062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc590100b90 0x7fc59019c960 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc58800b780 tx=0x7fc58800bd60 comp rx=0 tx=0).stop 2023-07-14T17:43:39.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.404+0000 7fc595dd9700 1 -- 172.21.15.88:0/3782149062 shutdown_connections 2023-07-14T17:43:39.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.404+0000 7fc595dd9700 1 --2- 172.21.15.88:0/3782149062 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc5780a1260 0x7fc5780a3710 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.404+0000 7fc595dd9700 1 --2- 172.21.15.88:0/3782149062 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc590100b90 0x7fc59019c960 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.404+0000 7fc595dd9700 1 --2- 172.21.15.88:0/3782149062 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc5900fe210 0x7fc590197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.404+0000 7fc595dd9700 1 --2- 172.21.15.88:0/3782149062 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc5900feca0 0x7fc5901984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:39.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.404+0000 7fc595dd9700 1 -- 172.21.15.88:0/3782149062 >> 172.21.15.88:0/3782149062 conn(0x7fc590075280 msgr2=0x7fc5900fbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:39.388 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.404+0000 7fc595dd9700 1 -- 172.21.15.88:0/3782149062 shutdown_connections 2023-07-14T17:43:39.389 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:39.404+0000 7fc595dd9700 1 -- 172.21.15.88:0/3782149062 wait complete. 2023-07-14T17:43:39.777 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743448 got 296352743450 for osd.10 2023-07-14T17:43:39.778 DEBUG:teuthology.parallel:result is None 2023-07-14T17:43:40.026 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:39 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3992290911' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T17:43:40.026 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:39 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/4217589333' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T17:43:40.026 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:39 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2434278703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T17:43:40.026 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:39 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3782149062' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T17:43:40.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:39 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3992290911' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T17:43:40.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:39 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/4217589333' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T17:43:40.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:39 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2434278703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T17:43:40.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:39 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3782149062' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T17:43:40.187 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135905 got 244813135907 for osd.8 2023-07-14T17:43:40.188 DEBUG:teuthology.parallel:result is None 2023-07-14T17:43:40.189 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182454 got 107374182457 for osd.3 2023-07-14T17:43:40.189 DEBUG:teuthology.parallel:result is None 2023-07-14T17:43:40.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:39 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3992290911' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T17:43:40.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:39 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/4217589333' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T17:43:40.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:39 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2434278703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T17:43:40.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:39 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3782149062' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T17:43:40.442 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411385 got 77309411388 for osd.2 2023-07-14T17:43:40.442 DEBUG:teuthology.parallel:result is None 2023-07-14T17:43:40.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.781+0000 7fa4b35f8700 1 -- 172.21.15.88:0/1279800528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4ac101150 msgr2=0x7fa4ac10bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:40.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.781+0000 7fa4b35f8700 1 --2- 172.21.15.88:0/1279800528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4ac101150 0x7fa4ac10bb40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400b0d0 tx=0x7fa4a400b3e0 comp rx=0 tx=0).stop 2023-07-14T17:43:40.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.781+0000 7fa4b35f8700 1 -- 172.21.15.88:0/1279800528 shutdown_connections 2023-07-14T17:43:40.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.781+0000 7fa4b35f8700 1 --2- 172.21.15.88:0/1279800528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4ac101150 0x7fa4ac10bb40 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:40.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.781+0000 7fa4b35f8700 1 --2- 172.21.15.88:0/1279800528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4ac0fef40 0x7fa4ac0ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:40.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.781+0000 7fa4b35f8700 1 --2- 172.21.15.88:0/1279800528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa4ac1007a0 0x7fa4ac100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:40.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.781+0000 7fa4b35f8700 1 -- 172.21.15.88:0/1279800528 >> 172.21.15.88:0/1279800528 conn(0x7fa4ac075c90 msgr2=0x7fa4ac076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:40.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.782+0000 7fa4b35f8700 1 -- 172.21.15.88:0/1279800528 shutdown_connections 2023-07-14T17:43:40.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.782+0000 7fa4b35f8700 1 -- 172.21.15.88:0/1279800528 wait complete. 2023-07-14T17:43:40.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.782+0000 7fa4b35f8700 1 Processor -- start 2023-07-14T17:43:40.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.782+0000 7fa4b35f8700 1 -- start start 2023-07-14T17:43:40.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.782+0000 7fa4b35f8700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa4ac0fef40 0x7fa4ac196c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:40.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.782+0000 7fa4b35f8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4ac1007a0 0x7fa4ac197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:40.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.782+0000 7fa4b35f8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4ac101150 0x7fa4ac1a1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:40.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.782+0000 7fa4b35f8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac197a80 con 0x7fa4ac1007a0 2023-07-14T17:43:40.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.782+0000 7fa4b35f8700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac1a21c0 con 0x7fa4ac101150 2023-07-14T17:43:40.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.782+0000 7fa4b35f8700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac1a2380 con 0x7fa4ac0fef40 2023-07-14T17:43:40.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.783+0000 7fa4b0b93700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4ac1007a0 0x7fa4ac197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:40.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.783+0000 7fa4b0b93700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4ac1007a0 0x7fa4ac197140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42414/0 (socket says 172.21.15.88:42414) 2023-07-14T17:43:40.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.783+0000 7fa4b0b93700 1 -- 172.21.15.88:0/33450039 learned_addr learned my addr 172.21.15.88:0/33450039 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:40.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.783+0000 7fa4b1b95700 1 --2- 172.21.15.88:0/33450039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4ac101150 0x7fa4ac1a1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:40.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.783+0000 7fa4b1394700 1 --2- 172.21.15.88:0/33450039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa4ac0fef40 0x7fa4ac196c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:40.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.783+0000 7fa4b0b93700 1 -- 172.21.15.88:0/33450039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa4ac0fef40 msgr2=0x7fa4ac196c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:40.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.783+0000 7fa4b0b93700 1 --2- 172.21.15.88:0/33450039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa4ac0fef40 0x7fa4ac196c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:40.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.783+0000 7fa4b0b93700 1 -- 172.21.15.88:0/33450039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4ac101150 msgr2=0x7fa4ac1a1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:40.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.783+0000 7fa4b0b93700 1 --2- 172.21.15.88:0/33450039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4ac101150 0x7fa4ac1a1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:40.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.783+0000 7fa4b0b93700 1 -- 172.21.15.88:0/33450039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a4009d00 con 0x7fa4ac1007a0 2023-07-14T17:43:40.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.783+0000 7fa4b1b95700 1 --2- 172.21.15.88:0/33450039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4ac101150 0x7fa4ac1a1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:43:40.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.783+0000 7fa4b1394700 1 --2- 172.21.15.88:0/33450039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa4ac0fef40 0x7fa4ac196c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:43:40.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.783+0000 7fa4b0b93700 1 --2- 172.21.15.88:0/33450039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4ac1007a0 0x7fa4ac197140 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00d980 tx=0x7fa49c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:40.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.783+0000 7fa4a27fc700 1 -- 172.21.15.88:0/33450039 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c00bb60 con 0x7fa4ac1007a0 2023-07-14T17:43:40.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.783+0000 7fa4b35f8700 1 -- 172.21.15.88:0/33450039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ac1a2660 con 0x7fa4ac1007a0 2023-07-14T17:43:40.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.783+0000 7fa4a27fc700 1 -- 172.21.15.88:0/33450039 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa49c007640 con 0x7fa4ac1007a0 2023-07-14T17:43:40.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.783+0000 7fa4a27fc700 1 -- 172.21.15.88:0/33450039 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49c004d60 con 0x7fa4ac1007a0 2023-07-14T17:43:40.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.783+0000 7fa4b35f8700 1 -- 172.21.15.88:0/33450039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ac1a2bb0 con 0x7fa4ac1007a0 2023-07-14T17:43:40.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.784+0000 7fa4b35f8700 1 -- 172.21.15.88:0/33450039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ac04e340 con 0x7fa4ac1007a0 2023-07-14T17:43:40.771 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.786+0000 7fa4a27fc700 1 -- 172.21.15.88:0/33450039 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa49c004020 con 0x7fa4ac1007a0 2023-07-14T17:43:40.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.787+0000 7fa4a27fc700 1 --2- 172.21.15.88:0/33450039 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa4980a11e0 0x7fa4980a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:40.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.787+0000 7fa4b1394700 1 --2- 172.21.15.88:0/33450039 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa4980a11e0 0x7fa4980a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:40.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.787+0000 7fa4a27fc700 1 -- 172.21.15.88:0/33450039 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa49c0cd080 con 0x7fa4ac1007a0 2023-07-14T17:43:40.772 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.787+0000 7fa4b1394700 1 --2- 172.21.15.88:0/33450039 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa4980a11e0 0x7fa4980a3690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8005fd0 tx=0x7fa4a8005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:40.773 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.790+0000 7fa4a27fc700 1 -- 172.21.15.88:0/33450039 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa49c096480 con 0x7fa4ac1007a0 2023-07-14T17:43:40.777 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789998 got 158913790000 for osd.5 2023-07-14T17:43:40.777 DEBUG:teuthology.parallel:result is None 2023-07-14T17:43:40.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.977+0000 7fa4b35f8700 1 -- 172.21.15.88:0/33450039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa4ac1a2dd0 con 0x7fa4ac1007a0 2023-07-14T17:43:40.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.978+0000 7fa4a27fc700 1 -- 172.21.15.88:0/33450039 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa49c016020 con 0x7fa4ac1007a0 2023-07-14T17:43:40.989 INFO:teuthology.orchestra.run.smithi088.stdout:34359738438 2023-07-14T17:43:40.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.980+0000 7fa4b35f8700 1 -- 172.21.15.88:0/33450039 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa4980a11e0 msgr2=0x7fa4980a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:40.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.980+0000 7fa4b35f8700 1 --2- 172.21.15.88:0/33450039 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa4980a11e0 0x7fa4980a3690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8005fd0 tx=0x7fa4a8005e30 comp rx=0 tx=0).stop 2023-07-14T17:43:40.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.980+0000 7fa4b35f8700 1 -- 172.21.15.88:0/33450039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4ac1007a0 msgr2=0x7fa4ac197140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:40.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.980+0000 7fa4b35f8700 1 --2- 172.21.15.88:0/33450039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4ac1007a0 0x7fa4ac197140 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00d980 tx=0x7fa49c00dd40 comp rx=0 tx=0).stop 2023-07-14T17:43:40.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.980+0000 7fa4b35f8700 1 -- 172.21.15.88:0/33450039 shutdown_connections 2023-07-14T17:43:40.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.980+0000 7fa4b35f8700 1 --2- 172.21.15.88:0/33450039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa4ac101150 0x7fa4ac1a1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:40.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.980+0000 7fa4b35f8700 1 --2- 172.21.15.88:0/33450039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa4ac0fef40 0x7fa4ac196c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:40.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.980+0000 7fa4b35f8700 1 --2- 172.21.15.88:0/33450039 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa4980a11e0 0x7fa4980a3690 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:40.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.980+0000 7fa4b35f8700 1 --2- 172.21.15.88:0/33450039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4ac1007a0 0x7fa4ac197140 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:40.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.980+0000 7fa4b35f8700 1 -- 172.21.15.88:0/33450039 >> 172.21.15.88:0/33450039 conn(0x7fa4ac075c90 msgr2=0x7fa4ac0fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:40.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.980+0000 7fa4b35f8700 1 -- 172.21.15.88:0/33450039 shutdown_connections 2023-07-14T17:43:40.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:40.980+0000 7fa4b35f8700 1 -- 172.21.15.88:0/33450039 wait complete. 2023-07-14T17:43:40.991 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:40 smithi088 ceph-mon[127465]: pgmap v284: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 8 op/s 2023-07-14T17:43:41.163 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:40 smithi178 ceph-mon[129368]: pgmap v284: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 8 op/s 2023-07-14T17:43:41.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:40 smithi107 ceph-mon[128999]: pgmap v284: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 8 op/s 2023-07-14T17:43:41.348 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738435 got 34359738438 for osd.0 2023-07-14T17:43:41.348 DEBUG:teuthology.parallel:result is None 2023-07-14T17:43:41.348 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-07-14T17:43:41.349 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T17:43:41.903 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:42.101 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:41 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/33450039' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T17:43:42.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:41 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/33450039' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T17:43:42.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:41 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/33450039' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T17:43:43.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.132+0000 7f5b8ae33700 1 -- 172.21.15.88:0/4275515884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b840fee40 msgr2=0x7f5b840ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:43.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.132+0000 7f5b8ae33700 1 --2- 172.21.15.88:0/4275515884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b840fee40 0x7f5b840ff210 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5b780099c0 tx=0x7f5b78009cd0 comp rx=0 tx=0).stop 2023-07-14T17:43:43.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.132+0000 7f5b8ae33700 1 -- 172.21.15.88:0/4275515884 shutdown_connections 2023-07-14T17:43:43.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.132+0000 7f5b8ae33700 1 --2- 172.21.15.88:0/4275515884 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b84103ee0 0x7f5b8410ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:43.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.132+0000 7f5b8ae33700 1 --2- 172.21.15.88:0/4275515884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b840fee40 0x7f5b840ff210 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:43.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.132+0000 7f5b8ae33700 1 --2- 172.21.15.88:0/4275515884 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b840ff7e0 0x7f5b841037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:43.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.132+0000 7f5b8ae33700 1 -- 172.21.15.88:0/4275515884 >> 172.21.15.88:0/4275515884 conn(0x7f5b84075ca0 msgr2=0x7f5b840760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:43.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.133+0000 7f5b8ae33700 1 -- 172.21.15.88:0/4275515884 shutdown_connections 2023-07-14T17:43:43.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.133+0000 7f5b8ae33700 1 -- 172.21.15.88:0/4275515884 wait complete. 2023-07-14T17:43:43.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.133+0000 7f5b8ae33700 1 Processor -- start 2023-07-14T17:43:43.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.133+0000 7f5b8ae33700 1 -- start start 2023-07-14T17:43:43.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b8ae33700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b840fee40 0x7f5b8419a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:43.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b8ae33700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b840ff7e0 0x7f5b8419af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:43.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b8ae33700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b84103ee0 0x7f5b84194a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:43.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b8ae33700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b841950b0 con 0x7f5b840fee40 2023-07-14T17:43:43.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b8ae33700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b84195220 con 0x7f5b84103ee0 2023-07-14T17:43:43.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b8ae33700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b841953e0 con 0x7f5b840ff7e0 2023-07-14T17:43:43.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b88bcf700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b840fee40 0x7f5b8419a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:43.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b88bcf700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b840fee40 0x7f5b8419a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42422/0 (socket says 172.21.15.88:42422) 2023-07-14T17:43:43.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b88bcf700 1 -- 172.21.15.88:0/3516121605 learned_addr learned my addr 172.21.15.88:0/3516121605 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:43.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b83fff700 1 --2- 172.21.15.88:0/3516121605 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b840ff7e0 0x7f5b8419af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:43.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b893d0700 1 --2- 172.21.15.88:0/3516121605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b84103ee0 0x7f5b84194a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:43.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b88bcf700 1 -- 172.21.15.88:0/3516121605 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b840ff7e0 msgr2=0x7f5b8419af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:43.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b88bcf700 1 --2- 172.21.15.88:0/3516121605 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b840ff7e0 0x7f5b8419af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:43.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b88bcf700 1 -- 172.21.15.88:0/3516121605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b84103ee0 msgr2=0x7f5b84194a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:43.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b88bcf700 1 --2- 172.21.15.88:0/3516121605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b84103ee0 0x7f5b84194a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:43.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b88bcf700 1 -- 172.21.15.88:0/3516121605 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b780095f0 con 0x7f5b840fee40 2023-07-14T17:43:43.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b893d0700 1 --2- 172.21.15.88:0/3516121605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b84103ee0 0x7f5b84194a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:43:43.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b88bcf700 1 --2- 172.21.15.88:0/3516121605 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b840fee40 0x7f5b8419a9c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5b78009f50 tx=0x7f5b78004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:43.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b81ffb700 1 -- 172.21.15.88:0/3516121605 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7801d070 con 0x7f5b840fee40 2023-07-14T17:43:43.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b81ffb700 1 -- 172.21.15.88:0/3516121605 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5b78003fc0 con 0x7f5b840fee40 2023-07-14T17:43:43.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b8ae33700 1 -- 172.21.15.88:0/3516121605 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b84195660 con 0x7f5b840fee40 2023-07-14T17:43:43.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.134+0000 7f5b81ffb700 1 -- 172.21.15.88:0/3516121605 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78022c30 con 0x7f5b840fee40 2023-07-14T17:43:43.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.135+0000 7f5b8ae33700 1 -- 172.21.15.88:0/3516121605 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b841a19b0 con 0x7f5b840fee40 2023-07-14T17:43:43.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.135+0000 7f5b8ae33700 1 -- 172.21.15.88:0/3516121605 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8404e340 con 0x7f5b840fee40 2023-07-14T17:43:43.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.137+0000 7f5b81ffb700 1 -- 172.21.15.88:0/3516121605 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5b78022470 con 0x7f5b840fee40 2023-07-14T17:43:43.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.137+0000 7f5b81ffb700 1 --2- 172.21.15.88:0/3516121605 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5b740a1260 0x7f5b740a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:43.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.137+0000 7f5b83fff700 1 --2- 172.21.15.88:0/3516121605 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5b740a1260 0x7f5b740a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:43.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.137+0000 7f5b81ffb700 1 -- 172.21.15.88:0/3516121605 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5b780ce190 con 0x7f5b840fee40 2023-07-14T17:43:43.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.138+0000 7f5b83fff700 1 --2- 172.21.15.88:0/3516121605 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5b740a1260 0x7f5b740a3710 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5b84077370 tx=0x7f5b7000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:43.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.141+0000 7f5b81ffb700 1 -- 172.21.15.88:0/3516121605 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5b780975c0 con 0x7f5b840fee40 2023-07-14T17:43:43.163 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:42 smithi178 ceph-mon[129368]: pgmap v285: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 10 op/s 2023-07-14T17:43:43.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:42 smithi088 ceph-mon[127465]: pgmap v285: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 10 op/s 2023-07-14T17:43:43.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:42 smithi107 ceph-mon[128999]: pgmap v285: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 10 op/s 2023-07-14T17:43:43.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.323+0000 7f5b8ae33700 1 -- 172.21.15.88:0/3516121605 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5b840fcc00 con 0x7f5b740a1260 2023-07-14T17:43:43.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.335+0000 7f5b81ffb700 1 -- 172.21.15.88:0/3516121605 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422653 (secure 0 0 0) 0x7f5b840fcc00 con 0x7f5b740a1260 2023-07-14T17:43:43.322 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:43:43.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.338+0000 7f5b8ae33700 1 -- 172.21.15.88:0/3516121605 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5b740a1260 msgr2=0x7f5b740a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:43.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.338+0000 7f5b8ae33700 1 --2- 172.21.15.88:0/3516121605 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5b740a1260 0x7f5b740a3710 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5b84077370 tx=0x7f5b7000b410 comp rx=0 tx=0).stop 2023-07-14T17:43:43.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.338+0000 7f5b8ae33700 1 -- 172.21.15.88:0/3516121605 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b840fee40 msgr2=0x7f5b8419a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:43.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.338+0000 7f5b8ae33700 1 --2- 172.21.15.88:0/3516121605 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b840fee40 0x7f5b8419a9c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5b78009f50 tx=0x7f5b78004710 comp rx=0 tx=0).stop 2023-07-14T17:43:43.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.339+0000 7f5b8ae33700 1 -- 172.21.15.88:0/3516121605 shutdown_connections 2023-07-14T17:43:43.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.339+0000 7f5b8ae33700 1 --2- 172.21.15.88:0/3516121605 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5b84103ee0 0x7f5b84194a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:43.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.339+0000 7f5b8ae33700 1 --2- 172.21.15.88:0/3516121605 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5b740a1260 0x7f5b740a3710 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:43.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.339+0000 7f5b8ae33700 1 --2- 172.21.15.88:0/3516121605 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5b840fee40 0x7f5b8419a9c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:43.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.339+0000 7f5b8ae33700 1 --2- 172.21.15.88:0/3516121605 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5b840ff7e0 0x7f5b8419af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:43.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.339+0000 7f5b8ae33700 1 -- 172.21.15.88:0/3516121605 >> 172.21.15.88:0/3516121605 conn(0x7f5b84075ca0 msgr2=0x7f5b840fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:43.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.339+0000 7f5b8ae33700 1 -- 172.21.15.88:0/3516121605 shutdown_connections 2023-07-14T17:43:43.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:43.339+0000 7f5b8ae33700 1 -- 172.21.15.88:0/3516121605 wait complete. 2023-07-14T17:43:43.326 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-07-14T17:43:43.967 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2023-07-14T17:43:41.886268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656175,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":519,"num_read_kb":376,"num_write":417,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":345,"ondisk_log_size":345,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337600,"kb_used_data":13408,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604224,"statfs":{"total":1151940427776,"available":1151594725376,"internally_reserved":0,"allocated":13729792,"data_stored":7543466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1652,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":128,"num_read_kb":63,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003611"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.060009+0000","last_change":"2023-07-14T17:42:42.540396+0000","last_active":"2023-07-14T17:42:48.060009+0000","last_peered":"2023-07-14T17:42:48.060009+0000","last_clean":"2023-07-14T17:42:48.060009+0000","last_became_active":"2023-07-14T17:42:42.539732+0000","last_became_peered":"2023-07-14T17:42:42.539732+0000","last_unstale":"2023-07-14T17:42:48.060009+0000","last_undegraded":"2023-07-14T17:42:48.060009+0000","last_fullsized":"2023-07-14T17:42:48.060009+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:15:44.925121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313254+0000","last_change":"2023-07-14T17:42:42.532153+0000","last_active":"2023-07-14T17:42:48.313254+0000","last_peered":"2023-07-14T17:42:48.313254+0000","last_clean":"2023-07-14T17:42:48.313254+0000","last_became_active":"2023-07-14T17:42:42.531745+0000","last_became_peered":"2023-07-14T17:42:42.531745+0000","last_unstale":"2023-07-14T17:42:48.313254+0000","last_undegraded":"2023-07-14T17:42:48.313254+0000","last_fullsized":"2023-07-14T17:42:48.313254+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:55.889251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313184+0000","last_change":"2023-07-14T17:42:42.549497+0000","last_active":"2023-07-14T17:42:48.313184+0000","last_peered":"2023-07-14T17:42:48.313184+0000","last_clean":"2023-07-14T17:42:48.313184+0000","last_became_active":"2023-07-14T17:42:42.549201+0000","last_became_peered":"2023-07-14T17:42:42.549201+0000","last_unstale":"2023-07-14T17:42:48.313184+0000","last_undegraded":"2023-07-14T17:42:48.313184+0000","last_fullsized":"2023-07-14T17:42:48.313184+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:20:16.806348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542180+0000","last_change":"2023-07-14T17:42:42.543460+0000","last_active":"2023-07-14T17:42:47.542180+0000","last_peered":"2023-07-14T17:42:47.542180+0000","last_clean":"2023-07-14T17:42:47.542180+0000","last_became_active":"2023-07-14T17:42:42.543127+0000","last_became_peered":"2023-07-14T17:42:42.543127+0000","last_unstale":"2023-07-14T17:42:47.542180+0000","last_undegraded":"2023-07-14T17:42:47.542180+0000","last_fullsized":"2023-07-14T17:42:47.542180+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:40:04.077779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835392+0000","last_change":"2023-07-14T17:42:42.534242+0000","last_active":"2023-07-14T17:42:47.835392+0000","last_peered":"2023-07-14T17:42:47.835392+0000","last_clean":"2023-07-14T17:42:47.835392+0000","last_became_active":"2023-07-14T17:42:42.533536+0000","last_became_peered":"2023-07-14T17:42:42.533536+0000","last_unstale":"2023-07-14T17:42:47.835392+0000","last_undegraded":"2023-07-14T17:42:47.835392+0000","last_fullsized":"2023-07-14T17:42:47.835392+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:57:39.927450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313767+0000","last_change":"2023-07-14T17:42:42.538042+0000","last_active":"2023-07-14T17:42:48.313767+0000","last_peered":"2023-07-14T17:42:48.313767+0000","last_clean":"2023-07-14T17:42:48.313767+0000","last_became_active":"2023-07-14T17:42:42.537612+0000","last_became_peered":"2023-07-14T17:42:42.537612+0000","last_unstale":"2023-07-14T17:42:48.313767+0000","last_undegraded":"2023-07-14T17:42:48.313767+0000","last_fullsized":"2023-07-14T17:42:48.313767+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:18:57.774815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313672+0000","last_change":"2023-07-14T17:42:42.545318+0000","last_active":"2023-07-14T17:42:48.313672+0000","last_peered":"2023-07-14T17:42:48.313672+0000","last_clean":"2023-07-14T17:42:48.313672+0000","last_became_active":"2023-07-14T17:42:42.544984+0000","last_became_peered":"2023-07-14T17:42:42.544984+0000","last_unstale":"2023-07-14T17:42:48.313672+0000","last_undegraded":"2023-07-14T17:42:48.313672+0000","last_fullsized":"2023-07-14T17:42:48.313672+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:21:01.756500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539255+0000","last_change":"2023-07-14T17:42:42.532609+0000","last_active":"2023-07-14T17:42:47.539255+0000","last_peered":"2023-07-14T17:42:47.539255+0000","last_clean":"2023-07-14T17:42:47.539255+0000","last_became_active":"2023-07-14T17:42:42.532270+0000","last_became_peered":"2023-07-14T17:42:42.532270+0000","last_unstale":"2023-07-14T17:42:47.539255+0000","last_undegraded":"2023-07-14T17:42:47.539255+0000","last_fullsized":"2023-07-14T17:42:47.539255+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:20:00.572052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.315239+0000","last_change":"2023-07-14T17:42:42.548973+0000","last_active":"2023-07-14T17:42:48.315239+0000","last_peered":"2023-07-14T17:42:48.315239+0000","last_clean":"2023-07-14T17:42:48.315239+0000","last_became_active":"2023-07-14T17:42:42.548672+0000","last_became_peered":"2023-07-14T17:42:42.548672+0000","last_unstale":"2023-07-14T17:42:48.315239+0000","last_undegraded":"2023-07-14T17:42:48.315239+0000","last_fullsized":"2023-07-14T17:42:48.315239+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:52:48.887371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.315082+0000","last_change":"2023-07-14T17:42:42.540969+0000","last_active":"2023-07-14T17:42:48.315082+0000","last_peered":"2023-07-14T17:42:48.315082+0000","last_clean":"2023-07-14T17:42:48.315082+0000","last_became_active":"2023-07-14T17:42:42.540653+0000","last_became_peered":"2023-07-14T17:42:42.540653+0000","last_unstale":"2023-07-14T17:42:48.315082+0000","last_undegraded":"2023-07-14T17:42:48.315082+0000","last_fullsized":"2023-07-14T17:42:48.315082+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:18:49.023990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314444+0000","last_change":"2023-07-14T17:42:42.550016+0000","last_active":"2023-07-14T17:42:48.314444+0000","last_peered":"2023-07-14T17:42:48.314444+0000","last_clean":"2023-07-14T17:42:48.314444+0000","last_became_active":"2023-07-14T17:42:42.549653+0000","last_became_peered":"2023-07-14T17:42:42.549653+0000","last_unstale":"2023-07-14T17:42:48.314444+0000","last_undegraded":"2023-07-14T17:42:48.314444+0000","last_fullsized":"2023-07-14T17:42:48.314444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:31:15.876030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542109+0000","last_change":"2023-07-14T17:42:42.552105+0000","last_active":"2023-07-14T17:42:47.542109+0000","last_peered":"2023-07-14T17:42:47.542109+0000","last_clean":"2023-07-14T17:42:47.542109+0000","last_became_active":"2023-07-14T17:42:42.550830+0000","last_became_peered":"2023-07-14T17:42:42.550830+0000","last_unstale":"2023-07-14T17:42:47.542109+0000","last_undegraded":"2023-07-14T17:42:47.542109+0000","last_fullsized":"2023-07-14T17:42:47.542109+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:47:37.896520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313576+0000","last_change":"2023-07-14T17:42:42.534963+0000","last_active":"2023-07-14T17:42:48.313576+0000","last_peered":"2023-07-14T17:42:48.313576+0000","last_clean":"2023-07-14T17:42:48.313576+0000","last_became_active":"2023-07-14T17:42:42.534546+0000","last_became_peered":"2023-07-14T17:42:42.534546+0000","last_unstale":"2023-07-14T17:42:48.313576+0000","last_undegraded":"2023-07-14T17:42:48.313576+0000","last_fullsized":"2023-07-14T17:42:48.313576+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:01:37.492549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.312996+0000","last_change":"2023-07-14T17:42:42.530073+0000","last_active":"2023-07-14T17:42:48.312996+0000","last_peered":"2023-07-14T17:42:48.312996+0000","last_clean":"2023-07-14T17:42:48.312996+0000","last_became_active":"2023-07-14T17:42:42.529767+0000","last_became_peered":"2023-07-14T17:42:42.529767+0000","last_unstale":"2023-07-14T17:42:48.312996+0000","last_undegraded":"2023-07-14T17:42:48.312996+0000","last_fullsized":"2023-07-14T17:42:48.312996+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:13:05.025890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.312868+0000","last_change":"2023-07-14T17:42:42.548168+0000","last_active":"2023-07-14T17:42:48.312868+0000","last_peered":"2023-07-14T17:42:48.312868+0000","last_clean":"2023-07-14T17:42:48.312868+0000","last_became_active":"2023-07-14T17:42:42.547363+0000","last_became_peered":"2023-07-14T17:42:42.547363+0000","last_unstale":"2023-07-14T17:42:48.312868+0000","last_undegraded":"2023-07-14T17:42:48.312868+0000","last_fullsized":"2023-07-14T17:42:48.312868+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:25:15.031613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313218+0000","last_change":"2023-07-14T17:42:42.551042+0000","last_active":"2023-07-14T17:42:48.313218+0000","last_peered":"2023-07-14T17:42:48.313218+0000","last_clean":"2023-07-14T17:42:48.313218+0000","last_became_active":"2023-07-14T17:42:42.550730+0000","last_became_peered":"2023-07-14T17:42:42.550730+0000","last_unstale":"2023-07-14T17:42:48.313218+0000","last_undegraded":"2023-07-14T17:42:48.313218+0000","last_fullsized":"2023-07-14T17:42:48.313218+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:34:57.949192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059890+0000","last_change":"2023-07-14T17:42:42.540528+0000","last_active":"2023-07-14T17:42:48.059890+0000","last_peered":"2023-07-14T17:42:48.059890+0000","last_clean":"2023-07-14T17:42:48.059890+0000","last_became_active":"2023-07-14T17:42:42.540093+0000","last_became_peered":"2023-07-14T17:42:42.540093+0000","last_unstale":"2023-07-14T17:42:48.059890+0000","last_undegraded":"2023-07-14T17:42:48.059890+0000","last_fullsized":"2023-07-14T17:42:48.059890+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:03:51.839812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833464+0000","last_change":"2023-07-14T17:42:42.539512+0000","last_active":"2023-07-14T17:42:47.833464+0000","last_peered":"2023-07-14T17:42:47.833464+0000","last_clean":"2023-07-14T17:42:47.833464+0000","last_became_active":"2023-07-14T17:42:42.539248+0000","last_became_peered":"2023-07-14T17:42:42.539248+0000","last_unstale":"2023-07-14T17:42:47.833464+0000","last_undegraded":"2023-07-14T17:42:47.833464+0000","last_fullsized":"2023-07-14T17:42:47.833464+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:02:47.593908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123669+0000","last_change":"2023-07-14T17:42:42.531630+0000","last_active":"2023-07-14T17:42:48.123669+0000","last_peered":"2023-07-14T17:42:48.123669+0000","last_clean":"2023-07-14T17:42:48.123669+0000","last_became_active":"2023-07-14T17:42:42.531333+0000","last_became_peered":"2023-07-14T17:42:42.531333+0000","last_unstale":"2023-07-14T17:42:48.123669+0000","last_undegraded":"2023-07-14T17:42:48.123669+0000","last_fullsized":"2023-07-14T17:42:48.123669+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:45:40.106913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123071+0000","last_change":"2023-07-14T17:42:46.543062+0000","last_active":"2023-07-14T17:42:48.123071+0000","last_peered":"2023-07-14T17:42:48.123071+0000","last_clean":"2023-07-14T17:42:48.123071+0000","last_became_active":"2023-07-14T17:42:46.542189+0000","last_became_peered":"2023-07-14T17:42:46.542189+0000","last_unstale":"2023-07-14T17:42:48.123071+0000","last_undegraded":"2023-07-14T17:42:48.123071+0000","last_fullsized":"2023-07-14T17:42:48.123071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:42:27.430887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":70,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:36.848252+0000","last_change":"2023-07-14T17:42:41.914589+0000","last_active":"2023-07-14T17:43:36.848252+0000","last_peered":"2023-07-14T17:43:36.848252+0000","last_clean":"2023-07-14T17:43:36.848252+0000","last_became_active":"2023-07-14T17:42:41.914275+0000","last_became_peered":"2023-07-14T17:42:41.914275+0000","last_unstale":"2023-07-14T17:43:36.848252+0000","last_undegraded":"2023-07-14T17:43:36.848252+0000","last_fullsized":"2023-07-14T17:43:36.848252+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:09:10.487178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":35,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540020+0000","last_change":"2023-07-14T17:42:42.537207+0000","last_active":"2023-07-14T17:42:47.540020+0000","last_peered":"2023-07-14T17:42:47.540020+0000","last_clean":"2023-07-14T17:42:47.540020+0000","last_became_active":"2023-07-14T17:42:42.536879+0000","last_became_peered":"2023-07-14T17:42:42.536879+0000","last_unstale":"2023-07-14T17:42:47.540020+0000","last_undegraded":"2023-07-14T17:42:47.540020+0000","last_fullsized":"2023-07-14T17:42:47.540020+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:09:29.936504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.541631+0000","last_change":"2023-07-14T17:42:46.542500+0000","last_active":"2023-07-14T17:42:47.541631+0000","last_peered":"2023-07-14T17:42:47.541631+0000","last_clean":"2023-07-14T17:42:47.541631+0000","last_became_active":"2023-07-14T17:42:46.541932+0000","last_became_peered":"2023-07-14T17:42:46.541932+0000","last_unstale":"2023-07-14T17:42:47.541631+0000","last_undegraded":"2023-07-14T17:42:47.541631+0000","last_fullsized":"2023-07-14T17:42:47.541631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:07:53.233373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540068+0000","last_change":"2023-07-14T17:42:41.918169+0000","last_active":"2023-07-14T17:42:47.540068+0000","last_peered":"2023-07-14T17:42:47.540068+0000","last_clean":"2023-07-14T17:42:47.540068+0000","last_became_active":"2023-07-14T17:42:41.917124+0000","last_became_peered":"2023-07-14T17:42:41.917124+0000","last_unstale":"2023-07-14T17:42:47.540068+0000","last_undegraded":"2023-07-14T17:42:47.540068+0000","last_fullsized":"2023-07-14T17:42:47.540068+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:23:35.325653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313826+0000","last_change":"2023-07-14T17:42:42.539265+0000","last_active":"2023-07-14T17:42:48.313826+0000","last_peered":"2023-07-14T17:42:48.313826+0000","last_clean":"2023-07-14T17:42:48.313826+0000","last_became_active":"2023-07-14T17:42:42.538933+0000","last_became_peered":"2023-07-14T17:42:42.538933+0000","last_unstale":"2023-07-14T17:42:48.313826+0000","last_undegraded":"2023-07-14T17:42:48.313826+0000","last_fullsized":"2023-07-14T17:42:48.313826+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:08.277561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542003+0000","last_change":"2023-07-14T17:42:46.550194+0000","last_active":"2023-07-14T17:42:47.542003+0000","last_peered":"2023-07-14T17:42:47.542003+0000","last_clean":"2023-07-14T17:42:47.542003+0000","last_became_active":"2023-07-14T17:42:46.549693+0000","last_became_peered":"2023-07-14T17:42:46.549693+0000","last_unstale":"2023-07-14T17:42:47.542003+0000","last_undegraded":"2023-07-14T17:42:47.542003+0000","last_fullsized":"2023-07-14T17:42:47.542003+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:15.448341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.781365+0000","last_change":"2023-07-14T17:42:41.918769+0000","last_active":"2023-07-14T17:43:04.781365+0000","last_peered":"2023-07-14T17:43:04.781365+0000","last_clean":"2023-07-14T17:43:04.781365+0000","last_became_active":"2023-07-14T17:42:41.918573+0000","last_became_peered":"2023-07-14T17:42:41.918573+0000","last_unstale":"2023-07-14T17:43:04.781365+0000","last_undegraded":"2023-07-14T17:43:04.781365+0000","last_fullsized":"2023-07-14T17:43:04.781365+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:19:21.409733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.315010+0000","last_change":"2023-07-14T17:42:42.543846+0000","last_active":"2023-07-14T17:42:48.315010+0000","last_peered":"2023-07-14T17:42:48.315010+0000","last_clean":"2023-07-14T17:42:48.315010+0000","last_became_active":"2023-07-14T17:42:42.543528+0000","last_became_peered":"2023-07-14T17:42:42.543528+0000","last_unstale":"2023-07-14T17:42:48.315010+0000","last_undegraded":"2023-07-14T17:42:48.315010+0000","last_fullsized":"2023-07-14T17:42:48.315010+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:12.999288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.541773+0000","last_change":"2023-07-14T17:42:46.547799+0000","last_active":"2023-07-14T17:42:47.541773+0000","last_peered":"2023-07-14T17:42:47.541773+0000","last_clean":"2023-07-14T17:42:47.541773+0000","last_became_active":"2023-07-14T17:42:46.547381+0000","last_became_peered":"2023-07-14T17:42:46.547381+0000","last_unstale":"2023-07-14T17:42:47.541773+0000","last_undegraded":"2023-07-14T17:42:47.541773+0000","last_fullsized":"2023-07-14T17:42:47.541773+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:46:15.220406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.771498+0000","last_change":"2023-07-14T17:42:41.914911+0000","last_active":"2023-07-14T17:43:03.771498+0000","last_peered":"2023-07-14T17:43:03.771498+0000","last_clean":"2023-07-14T17:43:03.771498+0000","last_became_active":"2023-07-14T17:42:41.914607+0000","last_became_peered":"2023-07-14T17:42:41.914607+0000","last_unstale":"2023-07-14T17:43:03.771498+0000","last_undegraded":"2023-07-14T17:43:03.771498+0000","last_fullsized":"2023-07-14T17:43:03.771498+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:45.771966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540287+0000","last_change":"2023-07-14T17:42:42.548866+0000","last_active":"2023-07-14T17:42:47.540287+0000","last_peered":"2023-07-14T17:42:47.540287+0000","last_clean":"2023-07-14T17:42:47.540287+0000","last_became_active":"2023-07-14T17:42:42.548562+0000","last_became_peered":"2023-07-14T17:42:42.548562+0000","last_unstale":"2023-07-14T17:42:47.540287+0000","last_undegraded":"2023-07-14T17:42:47.540287+0000","last_fullsized":"2023-07-14T17:42:47.540287+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:56:04.312858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123537+0000","last_change":"2023-07-14T17:42:46.538879+0000","last_active":"2023-07-14T17:42:48.123537+0000","last_peered":"2023-07-14T17:42:48.123537+0000","last_clean":"2023-07-14T17:42:48.123537+0000","last_became_active":"2023-07-14T17:42:46.538476+0000","last_became_peered":"2023-07-14T17:42:46.538476+0000","last_unstale":"2023-07-14T17:42:48.123537+0000","last_undegraded":"2023-07-14T17:42:48.123537+0000","last_fullsized":"2023-07-14T17:42:48.123537+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:40:22.866252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":66,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:38.835798+0000","last_change":"2023-07-14T17:42:41.917215+0000","last_active":"2023-07-14T17:43:38.835798+0000","last_peered":"2023-07-14T17:43:38.835798+0000","last_clean":"2023-07-14T17:43:38.835798+0000","last_became_active":"2023-07-14T17:42:41.916640+0000","last_became_peered":"2023-07-14T17:42:41.916640+0000","last_unstale":"2023-07-14T17:43:38.835798+0000","last_undegraded":"2023-07-14T17:43:38.835798+0000","last_fullsized":"2023-07-14T17:43:38.835798+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:37:10.013820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835389+0000","last_change":"2023-07-14T17:42:42.534347+0000","last_active":"2023-07-14T17:42:47.835389+0000","last_peered":"2023-07-14T17:42:47.835389+0000","last_clean":"2023-07-14T17:42:47.835389+0000","last_became_active":"2023-07-14T17:42:42.533958+0000","last_became_peered":"2023-07-14T17:42:42.533958+0000","last_unstale":"2023-07-14T17:42:47.835389+0000","last_undegraded":"2023-07-14T17:42:47.835389+0000","last_fullsized":"2023-07-14T17:42:47.835389+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:52:39.908230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123386+0000","last_change":"2023-07-14T17:42:46.543144+0000","last_active":"2023-07-14T17:42:48.123386+0000","last_peered":"2023-07-14T17:42:48.123386+0000","last_clean":"2023-07-14T17:42:48.123386+0000","last_became_active":"2023-07-14T17:42:46.542311+0000","last_became_peered":"2023-07-14T17:42:46.542311+0000","last_unstale":"2023-07-14T17:42:48.123386+0000","last_undegraded":"2023-07-14T17:42:48.123386+0000","last_fullsized":"2023-07-14T17:42:48.123386+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:05:15.245426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.762300+0000","last_change":"2023-07-14T17:42:41.916174+0000","last_active":"2023-07-14T17:43:02.762300+0000","last_peered":"2023-07-14T17:43:02.762300+0000","last_clean":"2023-07-14T17:43:02.762300+0000","last_became_active":"2023-07-14T17:42:41.915867+0000","last_became_peered":"2023-07-14T17:42:41.915867+0000","last_unstale":"2023-07-14T17:43:02.762300+0000","last_undegraded":"2023-07-14T17:43:02.762300+0000","last_fullsized":"2023-07-14T17:43:02.762300+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:43.000811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540038+0000","last_change":"2023-07-14T17:42:42.547064+0000","last_active":"2023-07-14T17:42:47.540038+0000","last_peered":"2023-07-14T17:42:47.540038+0000","last_clean":"2023-07-14T17:42:47.540038+0000","last_became_active":"2023-07-14T17:42:42.546777+0000","last_became_peered":"2023-07-14T17:42:42.546777+0000","last_unstale":"2023-07-14T17:42:47.540038+0000","last_undegraded":"2023-07-14T17:42:47.540038+0000","last_fullsized":"2023-07-14T17:42:47.540038+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:39:05.720215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059852+0000","last_change":"2023-07-14T17:42:46.548647+0000","last_active":"2023-07-14T17:42:48.059852+0000","last_peered":"2023-07-14T17:42:48.059852+0000","last_clean":"2023-07-14T17:42:48.059852+0000","last_became_active":"2023-07-14T17:42:46.548142+0000","last_became_peered":"2023-07-14T17:42:46.548142+0000","last_unstale":"2023-07-14T17:42:48.059852+0000","last_undegraded":"2023-07-14T17:42:48.059852+0000","last_fullsized":"2023-07-14T17:42:48.059852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:52:44.965835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:01.756003+0000","last_change":"2023-07-14T17:42:41.911675+0000","last_active":"2023-07-14T17:43:01.756003+0000","last_peered":"2023-07-14T17:43:01.756003+0000","last_clean":"2023-07-14T17:43:01.756003+0000","last_became_active":"2023-07-14T17:42:41.911299+0000","last_became_peered":"2023-07-14T17:42:41.911299+0000","last_unstale":"2023-07-14T17:43:01.756003+0000","last_undegraded":"2023-07-14T17:43:01.756003+0000","last_fullsized":"2023-07-14T17:43:01.756003+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:09.880954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539414+0000","last_change":"2023-07-14T17:42:42.536211+0000","last_active":"2023-07-14T17:42:47.539414+0000","last_peered":"2023-07-14T17:42:47.539414+0000","last_clean":"2023-07-14T17:42:47.539414+0000","last_became_active":"2023-07-14T17:42:42.535763+0000","last_became_peered":"2023-07-14T17:42:42.535763+0000","last_unstale":"2023-07-14T17:42:47.539414+0000","last_undegraded":"2023-07-14T17:42:47.539414+0000","last_fullsized":"2023-07-14T17:42:47.539414+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:49:14.800525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313179+0000","last_change":"2023-07-14T17:42:46.540917+0000","last_active":"2023-07-14T17:42:48.313179+0000","last_peered":"2023-07-14T17:42:48.313179+0000","last_clean":"2023-07-14T17:42:48.313179+0000","last_became_active":"2023-07-14T17:42:46.540575+0000","last_became_peered":"2023-07-14T17:42:46.540575+0000","last_unstale":"2023-07-14T17:42:48.313179+0000","last_undegraded":"2023-07-14T17:42:48.313179+0000","last_fullsized":"2023-07-14T17:42:48.313179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:02:20.835521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.771356+0000","last_change":"2023-07-14T17:42:41.917646+0000","last_active":"2023-07-14T17:43:03.771356+0000","last_peered":"2023-07-14T17:43:03.771356+0000","last_clean":"2023-07-14T17:43:03.771356+0000","last_became_active":"2023-07-14T17:42:41.917331+0000","last_became_peered":"2023-07-14T17:42:41.917331+0000","last_unstale":"2023-07-14T17:43:03.771356+0000","last_undegraded":"2023-07-14T17:43:03.771356+0000","last_fullsized":"2023-07-14T17:43:03.771356+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:41:16.425142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833094+0000","last_change":"2023-07-14T17:42:42.526696+0000","last_active":"2023-07-14T17:42:47.833094+0000","last_peered":"2023-07-14T17:42:47.833094+0000","last_clean":"2023-07-14T17:42:47.833094+0000","last_became_active":"2023-07-14T17:42:42.526353+0000","last_became_peered":"2023-07-14T17:42:42.526353+0000","last_unstale":"2023-07-14T17:42:47.833094+0000","last_undegraded":"2023-07-14T17:42:47.833094+0000","last_fullsized":"2023-07-14T17:42:47.833094+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:01:54.943614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539428+0000","last_change":"2023-07-14T17:42:46.545460+0000","last_active":"2023-07-14T17:42:47.539428+0000","last_peered":"2023-07-14T17:42:47.539428+0000","last_clean":"2023-07-14T17:42:47.539428+0000","last_became_active":"2023-07-14T17:42:46.544897+0000","last_became_peered":"2023-07-14T17:42:46.544897+0000","last_unstale":"2023-07-14T17:42:47.539428+0000","last_undegraded":"2023-07-14T17:42:47.539428+0000","last_fullsized":"2023-07-14T17:42:47.539428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:18:28.628297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":98,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:39.836808+0000","last_change":"2023-07-14T17:42:41.912266+0000","last_active":"2023-07-14T17:43:39.836808+0000","last_peered":"2023-07-14T17:43:39.836808+0000","last_clean":"2023-07-14T17:43:39.836808+0000","last_became_active":"2023-07-14T17:42:41.911944+0000","last_became_peered":"2023-07-14T17:42:41.911944+0000","last_unstale":"2023-07-14T17:43:39.836808+0000","last_undegraded":"2023-07-14T17:43:39.836808+0000","last_fullsized":"2023-07-14T17:43:39.836808+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:10:15.152078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":72,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.315346+0000","last_change":"2023-07-14T17:42:42.548609+0000","last_active":"2023-07-14T17:42:48.315346+0000","last_peered":"2023-07-14T17:42:48.315346+0000","last_clean":"2023-07-14T17:42:48.315346+0000","last_became_active":"2023-07-14T17:42:42.548253+0000","last_became_peered":"2023-07-14T17:42:42.548253+0000","last_unstale":"2023-07-14T17:42:48.315346+0000","last_undegraded":"2023-07-14T17:42:48.315346+0000","last_fullsized":"2023-07-14T17:42:48.315346+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:02:49.871094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833114+0000","last_change":"2023-07-14T17:42:46.547324+0000","last_active":"2023-07-14T17:42:47.833114+0000","last_peered":"2023-07-14T17:42:47.833114+0000","last_clean":"2023-07-14T17:42:47.833114+0000","last_became_active":"2023-07-14T17:42:46.546876+0000","last_became_peered":"2023-07-14T17:42:46.546876+0000","last_unstale":"2023-07-14T17:42:47.833114+0000","last_undegraded":"2023-07-14T17:42:47.833114+0000","last_fullsized":"2023-07-14T17:42:47.833114+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:30:13.764106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.772549+0000","last_change":"2023-07-14T17:42:41.919148+0000","last_active":"2023-07-14T17:43:03.772549+0000","last_peered":"2023-07-14T17:43:03.772549+0000","last_clean":"2023-07-14T17:43:03.772549+0000","last_became_active":"2023-07-14T17:42:41.918842+0000","last_became_peered":"2023-07-14T17:42:41.918842+0000","last_unstale":"2023-07-14T17:43:03.772549+0000","last_undegraded":"2023-07-14T17:43:03.772549+0000","last_fullsized":"2023-07-14T17:43:03.772549+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:04:34.077766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313896+0000","last_change":"2023-07-14T17:42:42.548081+0000","last_active":"2023-07-14T17:42:48.313896+0000","last_peered":"2023-07-14T17:42:48.313896+0000","last_clean":"2023-07-14T17:42:48.313896+0000","last_became_active":"2023-07-14T17:42:42.547876+0000","last_became_peered":"2023-07-14T17:42:42.547876+0000","last_unstale":"2023-07-14T17:42:48.313896+0000","last_undegraded":"2023-07-14T17:42:48.313896+0000","last_fullsized":"2023-07-14T17:42:48.313896+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:53:08.262819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542183+0000","last_change":"2023-07-14T17:42:46.550516+0000","last_active":"2023-07-14T17:42:47.542183+0000","last_peered":"2023-07-14T17:42:47.542183+0000","last_clean":"2023-07-14T17:42:47.542183+0000","last_became_active":"2023-07-14T17:42:46.549588+0000","last_became_peered":"2023-07-14T17:42:46.549588+0000","last_unstale":"2023-07-14T17:42:47.542183+0000","last_undegraded":"2023-07-14T17:42:47.542183+0000","last_fullsized":"2023-07-14T17:42:47.542183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:21.155958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:02.762568+0000","last_change":"2023-07-14T17:42:41.917527+0000","last_active":"2023-07-14T17:43:02.762568+0000","last_peered":"2023-07-14T17:43:02.762568+0000","last_clean":"2023-07-14T17:43:02.762568+0000","last_became_active":"2023-07-14T17:42:41.917239+0000","last_became_peered":"2023-07-14T17:42:41.917239+0000","last_unstale":"2023-07-14T17:43:02.762568+0000","last_undegraded":"2023-07-14T17:43:02.762568+0000","last_fullsized":"2023-07-14T17:43:02.762568+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:04:18.619214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.315351+0000","last_change":"2023-07-14T17:42:42.554866+0000","last_active":"2023-07-14T17:42:48.315351+0000","last_peered":"2023-07-14T17:42:48.315351+0000","last_clean":"2023-07-14T17:42:48.315351+0000","last_became_active":"2023-07-14T17:42:42.554585+0000","last_became_peered":"2023-07-14T17:42:42.554585+0000","last_unstale":"2023-07-14T17:42:48.315351+0000","last_undegraded":"2023-07-14T17:42:48.315351+0000","last_fullsized":"2023-07-14T17:42:48.315351+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:01.732200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833167+0000","last_change":"2023-07-14T17:42:46.544788+0000","last_active":"2023-07-14T17:42:47.833167+0000","last_peered":"2023-07-14T17:42:47.833167+0000","last_clean":"2023-07-14T17:42:47.833167+0000","last_became_active":"2023-07-14T17:42:46.544419+0000","last_became_peered":"2023-07-14T17:42:46.544419+0000","last_unstale":"2023-07-14T17:42:47.833167+0000","last_undegraded":"2023-07-14T17:42:47.833167+0000","last_fullsized":"2023-07-14T17:42:47.833167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:03:38.940614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"86'33","reported_seq":133,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:39.836808+0000","last_change":"2023-07-14T17:42:41.917069+0000","last_active":"2023-07-14T17:43:39.836808+0000","last_peered":"2023-07-14T17:43:39.836808+0000","last_clean":"2023-07-14T17:43:39.836808+0000","last_became_active":"2023-07-14T17:42:41.916630+0000","last_became_peered":"2023-07-14T17:42:41.916630+0000","last_unstale":"2023-07-14T17:43:39.836808+0000","last_undegraded":"2023-07-14T17:43:39.836808+0000","last_fullsized":"2023-07-14T17:43:39.836808+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:05:57.979299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58551,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":78,"num_read_kb":95,"num_write":35,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835769+0000","last_change":"2023-07-14T17:42:42.542124+0000","last_active":"2023-07-14T17:42:47.835769+0000","last_peered":"2023-07-14T17:42:47.835769+0000","last_clean":"2023-07-14T17:42:47.835769+0000","last_became_active":"2023-07-14T17:42:42.541851+0000","last_became_peered":"2023-07-14T17:42:42.541851+0000","last_unstale":"2023-07-14T17:42:47.835769+0000","last_undegraded":"2023-07-14T17:42:47.835769+0000","last_fullsized":"2023-07-14T17:42:47.835769+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:06.253283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540680+0000","last_change":"2023-07-14T17:42:46.546167+0000","last_active":"2023-07-14T17:42:47.540680+0000","last_peered":"2023-07-14T17:42:47.540680+0000","last_clean":"2023-07-14T17:42:47.540680+0000","last_became_active":"2023-07-14T17:42:46.545768+0000","last_became_peered":"2023-07-14T17:42:46.545768+0000","last_unstale":"2023-07-14T17:42:47.540680+0000","last_undegraded":"2023-07-14T17:42:47.540680+0000","last_fullsized":"2023-07-14T17:42:47.540680+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:35:53.682315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.781459+0000","last_change":"2023-07-14T17:42:41.918858+0000","last_active":"2023-07-14T17:43:04.781459+0000","last_peered":"2023-07-14T17:43:04.781459+0000","last_clean":"2023-07-14T17:43:04.781459+0000","last_became_active":"2023-07-14T17:42:41.918568+0000","last_became_peered":"2023-07-14T17:42:41.918568+0000","last_unstale":"2023-07-14T17:43:04.781459+0000","last_undegraded":"2023-07-14T17:43:04.781459+0000","last_fullsized":"2023-07-14T17:43:04.781459+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:14:24.080564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313819+0000","last_change":"2023-07-14T17:42:42.542310+0000","last_active":"2023-07-14T17:42:48.313819+0000","last_peered":"2023-07-14T17:42:48.313819+0000","last_clean":"2023-07-14T17:42:48.313819+0000","last_became_active":"2023-07-14T17:42:42.541989+0000","last_became_peered":"2023-07-14T17:42:42.541989+0000","last_unstale":"2023-07-14T17:42:48.313819+0000","last_undegraded":"2023-07-14T17:42:48.313819+0000","last_fullsized":"2023-07-14T17:42:48.313819+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:41:45.629010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059652+0000","last_change":"2023-07-14T17:42:46.548737+0000","last_active":"2023-07-14T17:42:48.059652+0000","last_peered":"2023-07-14T17:42:48.059652+0000","last_clean":"2023-07-14T17:42:48.059652+0000","last_became_active":"2023-07-14T17:42:46.548255+0000","last_became_peered":"2023-07-14T17:42:46.548255+0000","last_unstale":"2023-07-14T17:42:48.059652+0000","last_undegraded":"2023-07-14T17:42:48.059652+0000","last_fullsized":"2023-07-14T17:42:48.059652+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:10.311183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.770445+0000","last_change":"2023-07-14T17:42:41.916382+0000","last_active":"2023-07-14T17:43:03.770445+0000","last_peered":"2023-07-14T17:43:03.770445+0000","last_clean":"2023-07-14T17:43:03.770445+0000","last_became_active":"2023-07-14T17:42:41.916082+0000","last_became_peered":"2023-07-14T17:42:41.916082+0000","last_unstale":"2023-07-14T17:43:03.770445+0000","last_undegraded":"2023-07-14T17:43:03.770445+0000","last_fullsized":"2023-07-14T17:43:03.770445+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:54:15.440758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313921+0000","last_change":"2023-07-14T17:42:42.546812+0000","last_active":"2023-07-14T17:42:48.313921+0000","last_peered":"2023-07-14T17:42:48.313921+0000","last_clean":"2023-07-14T17:42:48.313921+0000","last_became_active":"2023-07-14T17:42:42.546517+0000","last_became_peered":"2023-07-14T17:42:42.546517+0000","last_unstale":"2023-07-14T17:42:48.313921+0000","last_undegraded":"2023-07-14T17:42:48.313921+0000","last_fullsized":"2023-07-14T17:42:48.313921+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:41:22.699813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833687+0000","last_change":"2023-07-14T17:42:46.547726+0000","last_active":"2023-07-14T17:42:47.833687+0000","last_peered":"2023-07-14T17:42:47.833687+0000","last_clean":"2023-07-14T17:42:47.833687+0000","last_became_active":"2023-07-14T17:42:46.547207+0000","last_became_peered":"2023-07-14T17:42:46.547207+0000","last_unstale":"2023-07-14T17:42:47.833687+0000","last_undegraded":"2023-07-14T17:42:47.833687+0000","last_fullsized":"2023-07-14T17:42:47.833687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:55:16.418869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"86'6","reported_seq":64,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:36.841894+0000","last_change":"2023-07-14T17:42:41.913802+0000","last_active":"2023-07-14T17:43:36.841894+0000","last_peered":"2023-07-14T17:43:36.841894+0000","last_clean":"2023-07-14T17:43:36.841894+0000","last_became_active":"2023-07-14T17:42:41.913467+0000","last_became_peered":"2023-07-14T17:42:41.913467+0000","last_unstale":"2023-07-14T17:43:36.841894+0000","last_undegraded":"2023-07-14T17:43:36.841894+0000","last_fullsized":"2023-07-14T17:43:36.841894+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:46:59.725226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313504+0000","last_change":"2023-07-14T17:42:42.545379+0000","last_active":"2023-07-14T17:42:48.313504+0000","last_peered":"2023-07-14T17:42:48.313504+0000","last_clean":"2023-07-14T17:42:48.313504+0000","last_became_active":"2023-07-14T17:42:42.545012+0000","last_became_peered":"2023-07-14T17:42:42.545012+0000","last_unstale":"2023-07-14T17:42:48.313504+0000","last_undegraded":"2023-07-14T17:42:48.313504+0000","last_fullsized":"2023-07-14T17:42:48.313504+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:10.885034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313517+0000","last_change":"2023-07-14T17:42:46.544524+0000","last_active":"2023-07-14T17:42:48.313517+0000","last_peered":"2023-07-14T17:42:48.313517+0000","last_clean":"2023-07-14T17:42:48.313517+0000","last_became_active":"2023-07-14T17:42:46.544152+0000","last_became_peered":"2023-07-14T17:42:46.544152+0000","last_unstale":"2023-07-14T17:42:48.313517+0000","last_undegraded":"2023-07-14T17:42:48.313517+0000","last_fullsized":"2023-07-14T17:42:48.313517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:01:14.398239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.771510+0000","last_change":"2023-07-14T17:42:41.917282+0000","last_active":"2023-07-14T17:43:03.771510+0000","last_peered":"2023-07-14T17:43:03.771510+0000","last_clean":"2023-07-14T17:43:03.771510+0000","last_became_active":"2023-07-14T17:42:41.916994+0000","last_became_peered":"2023-07-14T17:42:41.916994+0000","last_unstale":"2023-07-14T17:43:03.771510+0000","last_undegraded":"2023-07-14T17:43:03.771510+0000","last_fullsized":"2023-07-14T17:43:03.771510+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:41:27.420237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540359+0000","last_change":"2023-07-14T17:42:42.548525+0000","last_active":"2023-07-14T17:42:47.540359+0000","last_peered":"2023-07-14T17:42:47.540359+0000","last_clean":"2023-07-14T17:42:47.540359+0000","last_became_active":"2023-07-14T17:42:42.548270+0000","last_became_peered":"2023-07-14T17:42:42.548270+0000","last_unstale":"2023-07-14T17:42:47.540359+0000","last_undegraded":"2023-07-14T17:42:47.540359+0000","last_fullsized":"2023-07-14T17:42:47.540359+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:18:32.383297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540365+0000","last_change":"2023-07-14T17:42:46.546480+0000","last_active":"2023-07-14T17:42:47.540365+0000","last_peered":"2023-07-14T17:42:47.540365+0000","last_clean":"2023-07-14T17:42:47.540365+0000","last_became_active":"2023-07-14T17:42:46.546078+0000","last_became_peered":"2023-07-14T17:42:46.546078+0000","last_unstale":"2023-07-14T17:42:47.540365+0000","last_undegraded":"2023-07-14T17:42:47.540365+0000","last_fullsized":"2023-07-14T17:42:47.540365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:17:38.547810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.781649+0000","last_change":"2023-07-14T17:42:41.914226+0000","last_active":"2023-07-14T17:43:04.781649+0000","last_peered":"2023-07-14T17:43:04.781649+0000","last_clean":"2023-07-14T17:43:04.781649+0000","last_became_active":"2023-07-14T17:42:41.913820+0000","last_became_peered":"2023-07-14T17:42:41.913820+0000","last_unstale":"2023-07-14T17:43:04.781649+0000","last_undegraded":"2023-07-14T17:43:04.781649+0000","last_fullsized":"2023-07-14T17:43:04.781649+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:31:36.040725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059864+0000","last_change":"2023-07-14T17:42:42.543549+0000","last_active":"2023-07-14T17:42:48.059864+0000","last_peered":"2023-07-14T17:42:48.059864+0000","last_clean":"2023-07-14T17:42:48.059864+0000","last_became_active":"2023-07-14T17:42:42.543245+0000","last_became_peered":"2023-07-14T17:42:42.543245+0000","last_unstale":"2023-07-14T17:42:48.059864+0000","last_undegraded":"2023-07-14T17:42:48.059864+0000","last_fullsized":"2023-07-14T17:42:48.059864+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:30:07.889057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835441+0000","last_change":"2023-07-14T17:42:46.577245+0000","last_active":"2023-07-14T17:42:47.835441+0000","last_peered":"2023-07-14T17:42:47.835441+0000","last_clean":"2023-07-14T17:42:47.835441+0000","last_became_active":"2023-07-14T17:42:46.576793+0000","last_became_peered":"2023-07-14T17:42:46.576793+0000","last_unstale":"2023-07-14T17:42:47.835441+0000","last_undegraded":"2023-07-14T17:42:47.835441+0000","last_fullsized":"2023-07-14T17:42:47.835441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:29:32.688546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833389+0000","last_change":"2023-07-14T17:42:41.917199+0000","last_active":"2023-07-14T17:42:47.833389+0000","last_peered":"2023-07-14T17:42:47.833389+0000","last_clean":"2023-07-14T17:42:47.833389+0000","last_became_active":"2023-07-14T17:42:41.916924+0000","last_became_peered":"2023-07-14T17:42:41.916924+0000","last_unstale":"2023-07-14T17:42:47.833389+0000","last_undegraded":"2023-07-14T17:42:47.833389+0000","last_fullsized":"2023-07-14T17:42:47.833389+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:12:02.515835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314451+0000","last_change":"2023-07-14T17:42:42.540531+0000","last_active":"2023-07-14T17:42:48.314451+0000","last_peered":"2023-07-14T17:42:48.314451+0000","last_clean":"2023-07-14T17:42:48.314451+0000","last_became_active":"2023-07-14T17:42:42.540165+0000","last_became_peered":"2023-07-14T17:42:42.540165+0000","last_unstale":"2023-07-14T17:42:48.314451+0000","last_undegraded":"2023-07-14T17:42:48.314451+0000","last_fullsized":"2023-07-14T17:42:48.314451+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:34:21.411165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313334+0000","last_change":"2023-07-14T17:42:46.537158+0000","last_active":"2023-07-14T17:42:48.313334+0000","last_peered":"2023-07-14T17:42:48.313334+0000","last_clean":"2023-07-14T17:42:48.313334+0000","last_became_active":"2023-07-14T17:42:46.536816+0000","last_became_peered":"2023-07-14T17:42:46.536816+0000","last_unstale":"2023-07-14T17:42:48.313334+0000","last_undegraded":"2023-07-14T17:42:48.313334+0000","last_fullsized":"2023-07-14T17:42:48.313334+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:47:53.565328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":65,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:39.852221+0000","last_change":"2023-07-14T17:42:41.913638+0000","last_active":"2023-07-14T17:43:39.852221+0000","last_peered":"2023-07-14T17:43:39.852221+0000","last_clean":"2023-07-14T17:43:39.852221+0000","last_became_active":"2023-07-14T17:42:41.913361+0000","last_became_peered":"2023-07-14T17:42:41.913361+0000","last_unstale":"2023-07-14T17:43:39.852221+0000","last_undegraded":"2023-07-14T17:43:39.852221+0000","last_fullsized":"2023-07-14T17:43:39.852221+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:30:25.777267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314326+0000","last_change":"2023-07-14T17:42:42.541738+0000","last_active":"2023-07-14T17:42:48.314326+0000","last_peered":"2023-07-14T17:42:48.314326+0000","last_clean":"2023-07-14T17:42:48.314326+0000","last_became_active":"2023-07-14T17:42:42.541264+0000","last_became_peered":"2023-07-14T17:42:42.541264+0000","last_unstale":"2023-07-14T17:42:48.314326+0000","last_undegraded":"2023-07-14T17:42:48.314326+0000","last_fullsized":"2023-07-14T17:42:48.314326+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:38:28.774135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313008+0000","last_change":"2023-07-14T17:42:46.576999+0000","last_active":"2023-07-14T17:42:48.313008+0000","last_peered":"2023-07-14T17:42:48.313008+0000","last_clean":"2023-07-14T17:42:48.313008+0000","last_became_active":"2023-07-14T17:42:46.576652+0000","last_became_peered":"2023-07-14T17:42:46.576652+0000","last_unstale":"2023-07-14T17:42:48.313008+0000","last_undegraded":"2023-07-14T17:42:48.313008+0000","last_fullsized":"2023-07-14T17:42:48.313008+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:59:23.913753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540039+0000","last_change":"2023-07-14T17:42:42.547053+0000","last_active":"2023-07-14T17:42:47.540039+0000","last_peered":"2023-07-14T17:42:47.540039+0000","last_clean":"2023-07-14T17:42:47.540039+0000","last_became_active":"2023-07-14T17:42:42.546735+0000","last_became_peered":"2023-07-14T17:42:42.546735+0000","last_unstale":"2023-07-14T17:42:47.540039+0000","last_undegraded":"2023-07-14T17:42:47.540039+0000","last_fullsized":"2023-07-14T17:42:47.540039+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:19:26.849479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":64,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:38.844426+0000","last_change":"2023-07-14T17:42:41.918692+0000","last_active":"2023-07-14T17:43:38.844426+0000","last_peered":"2023-07-14T17:43:38.844426+0000","last_clean":"2023-07-14T17:43:38.844426+0000","last_became_active":"2023-07-14T17:42:41.918360+0000","last_became_peered":"2023-07-14T17:42:41.918360+0000","last_unstale":"2023-07-14T17:43:38.844426+0000","last_undegraded":"2023-07-14T17:43:38.844426+0000","last_fullsized":"2023-07-14T17:43:38.844426+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:40:45.927843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314271+0000","last_change":"2023-07-14T17:42:46.543997+0000","last_active":"2023-07-14T17:42:48.314271+0000","last_peered":"2023-07-14T17:42:48.314271+0000","last_clean":"2023-07-14T17:42:48.314271+0000","last_became_active":"2023-07-14T17:42:46.543650+0000","last_became_peered":"2023-07-14T17:42:46.543650+0000","last_unstale":"2023-07-14T17:42:48.314271+0000","last_undegraded":"2023-07-14T17:42:48.314271+0000","last_fullsized":"2023-07-14T17:42:48.314271+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:25:35.518988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539400+0000","last_change":"2023-07-14T17:42:42.548860+0000","last_active":"2023-07-14T17:42:47.539400+0000","last_peered":"2023-07-14T17:42:47.539400+0000","last_clean":"2023-07-14T17:42:47.539400+0000","last_became_active":"2023-07-14T17:42:42.548564+0000","last_became_peered":"2023-07-14T17:42:42.548564+0000","last_unstale":"2023-07-14T17:42:47.539400+0000","last_undegraded":"2023-07-14T17:42:47.539400+0000","last_fullsized":"2023-07-14T17:42:47.539400+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:07:45.387117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.782501+0000","last_change":"2023-07-14T17:42:41.912601+0000","last_active":"2023-07-14T17:43:04.782501+0000","last_peered":"2023-07-14T17:43:04.782501+0000","last_clean":"2023-07-14T17:43:04.782501+0000","last_became_active":"2023-07-14T17:42:41.912265+0000","last_became_peered":"2023-07-14T17:42:41.912265+0000","last_unstale":"2023-07-14T17:43:04.782501+0000","last_undegraded":"2023-07-14T17:43:04.782501+0000","last_fullsized":"2023-07-14T17:43:04.782501+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:58.872606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":343,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.312950+0000","last_change":"2023-07-14T17:41:43.696093+0000","last_active":"2023-07-14T17:42:48.312950+0000","last_peered":"2023-07-14T17:42:48.312950+0000","last_clean":"2023-07-14T17:42:48.312950+0000","last_became_active":"2023-07-14T17:41:43.695511+0000","last_became_peered":"2023-07-14T17:41:43.695511+0000","last_unstale":"2023-07-14T17:42:48.312950+0000","last_undegraded":"2023-07-14T17:42:48.312950+0000","last_fullsized":"2023-07-14T17:42:48.312950+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:38:50.657865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:38:50.657865+0000","last_clean_scrub_stamp":"2023-07-14T17:38:50.657865+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:30:55.242715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540596+0000","last_change":"2023-07-14T17:42:46.577540+0000","last_active":"2023-07-14T17:42:47.540596+0000","last_peered":"2023-07-14T17:42:47.540596+0000","last_clean":"2023-07-14T17:42:47.540596+0000","last_became_active":"2023-07-14T17:42:46.577067+0000","last_became_peered":"2023-07-14T17:42:46.577067+0000","last_unstale":"2023-07-14T17:42:47.540596+0000","last_undegraded":"2023-07-14T17:42:47.540596+0000","last_fullsized":"2023-07-14T17:42:47.540596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:39:25.538930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313453+0000","last_change":"2023-07-14T17:42:42.543243+0000","last_active":"2023-07-14T17:42:48.313453+0000","last_peered":"2023-07-14T17:42:48.313453+0000","last_clean":"2023-07-14T17:42:48.313453+0000","last_became_active":"2023-07-14T17:42:42.542965+0000","last_became_peered":"2023-07-14T17:42:42.542965+0000","last_unstale":"2023-07-14T17:42:48.313453+0000","last_undegraded":"2023-07-14T17:42:48.313453+0000","last_fullsized":"2023-07-14T17:42:48.313453+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:08:21.008138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:01.753945+0000","last_change":"2023-07-14T17:42:41.920806+0000","last_active":"2023-07-14T17:43:01.753945+0000","last_peered":"2023-07-14T17:43:01.753945+0000","last_clean":"2023-07-14T17:43:01.753945+0000","last_became_active":"2023-07-14T17:42:41.920528+0000","last_became_peered":"2023-07-14T17:42:41.920528+0000","last_unstale":"2023-07-14T17:43:01.753945+0000","last_undegraded":"2023-07-14T17:43:01.753945+0000","last_fullsized":"2023-07-14T17:43:01.753945+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:43:43.408841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059657+0000","last_change":"2023-07-14T17:42:46.542026+0000","last_active":"2023-07-14T17:42:48.059657+0000","last_peered":"2023-07-14T17:42:48.059657+0000","last_clean":"2023-07-14T17:42:48.059657+0000","last_became_active":"2023-07-14T17:42:46.541672+0000","last_became_peered":"2023-07-14T17:42:46.541672+0000","last_unstale":"2023-07-14T17:42:48.059657+0000","last_undegraded":"2023-07-14T17:42:48.059657+0000","last_fullsized":"2023-07-14T17:42:48.059657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:45:35.131993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.772096+0000","last_change":"2023-07-14T17:42:41.915798+0000","last_active":"2023-07-14T17:43:03.772096+0000","last_peered":"2023-07-14T17:43:03.772096+0000","last_clean":"2023-07-14T17:43:03.772096+0000","last_became_active":"2023-07-14T17:42:41.915492+0000","last_became_peered":"2023-07-14T17:42:41.915492+0000","last_unstale":"2023-07-14T17:43:03.772096+0000","last_undegraded":"2023-07-14T17:43:03.772096+0000","last_fullsized":"2023-07-14T17:43:03.772096+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:25:42.031839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313564+0000","last_change":"2023-07-14T17:42:42.533500+0000","last_active":"2023-07-14T17:42:48.313564+0000","last_peered":"2023-07-14T17:42:48.313564+0000","last_clean":"2023-07-14T17:42:48.313564+0000","last_became_active":"2023-07-14T17:42:42.533232+0000","last_became_peered":"2023-07-14T17:42:42.533232+0000","last_unstale":"2023-07-14T17:42:48.313564+0000","last_undegraded":"2023-07-14T17:42:48.313564+0000","last_fullsized":"2023-07-14T17:42:48.313564+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:57:14.212427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539366+0000","last_change":"2023-07-14T17:42:46.577131+0000","last_active":"2023-07-14T17:42:47.539366+0000","last_peered":"2023-07-14T17:42:47.539366+0000","last_clean":"2023-07-14T17:42:47.539366+0000","last_became_active":"2023-07-14T17:42:46.576695+0000","last_became_peered":"2023-07-14T17:42:46.576695+0000","last_unstale":"2023-07-14T17:42:47.539366+0000","last_undegraded":"2023-07-14T17:42:47.539366+0000","last_fullsized":"2023-07-14T17:42:47.539366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:49:05.554462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'9","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:07.833208+0000","last_change":"2023-07-14T17:42:41.919021+0000","last_active":"2023-07-14T17:43:07.833208+0000","last_peered":"2023-07-14T17:43:07.833208+0000","last_clean":"2023-07-14T17:43:07.833208+0000","last_became_active":"2023-07-14T17:42:41.918714+0000","last_became_peered":"2023-07-14T17:42:41.918714+0000","last_unstale":"2023-07-14T17:43:07.833208+0000","last_undegraded":"2023-07-14T17:43:07.833208+0000","last_fullsized":"2023-07-14T17:43:07.833208+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:40:48.400679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539836+0000","last_change":"2023-07-14T17:42:42.534661+0000","last_active":"2023-07-14T17:42:47.539836+0000","last_peered":"2023-07-14T17:42:47.539836+0000","last_clean":"2023-07-14T17:42:47.539836+0000","last_became_active":"2023-07-14T17:42:42.534357+0000","last_became_peered":"2023-07-14T17:42:42.534357+0000","last_unstale":"2023-07-14T17:42:47.539836+0000","last_undegraded":"2023-07-14T17:42:47.539836+0000","last_fullsized":"2023-07-14T17:42:47.539836+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:24:53.645011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.122946+0000","last_change":"2023-07-14T17:42:46.538648+0000","last_active":"2023-07-14T17:42:48.122946+0000","last_peered":"2023-07-14T17:42:48.122946+0000","last_clean":"2023-07-14T17:42:48.122946+0000","last_became_active":"2023-07-14T17:42:46.538225+0000","last_became_peered":"2023-07-14T17:42:46.538225+0000","last_unstale":"2023-07-14T17:42:48.122946+0000","last_undegraded":"2023-07-14T17:42:48.122946+0000","last_fullsized":"2023-07-14T17:42:48.122946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:36:57.320814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.772287+0000","last_change":"2023-07-14T17:42:41.915406+0000","last_active":"2023-07-14T17:43:03.772287+0000","last_peered":"2023-07-14T17:43:03.772287+0000","last_clean":"2023-07-14T17:43:03.772287+0000","last_became_active":"2023-07-14T17:42:41.914967+0000","last_became_peered":"2023-07-14T17:42:41.914967+0000","last_unstale":"2023-07-14T17:43:03.772287+0000","last_undegraded":"2023-07-14T17:43:03.772287+0000","last_fullsized":"2023-07-14T17:43:03.772287+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:56:07.113627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313341+0000","last_change":"2023-07-14T17:42:42.539052+0000","last_active":"2023-07-14T17:42:48.313341+0000","last_peered":"2023-07-14T17:42:48.313341+0000","last_clean":"2023-07-14T17:42:48.313341+0000","last_became_active":"2023-07-14T17:42:42.538752+0000","last_became_peered":"2023-07-14T17:42:42.538752+0000","last_unstale":"2023-07-14T17:42:48.313341+0000","last_undegraded":"2023-07-14T17:42:48.313341+0000","last_fullsized":"2023-07-14T17:42:48.313341+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:25:31.544906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542387+0000","last_change":"2023-07-14T17:42:46.577224+0000","last_active":"2023-07-14T17:42:47.542387+0000","last_peered":"2023-07-14T17:42:47.542387+0000","last_clean":"2023-07-14T17:42:47.542387+0000","last_became_active":"2023-07-14T17:42:46.576781+0000","last_became_peered":"2023-07-14T17:42:46.576781+0000","last_unstale":"2023-07-14T17:42:47.542387+0000","last_undegraded":"2023-07-14T17:42:47.542387+0000","last_fullsized":"2023-07-14T17:42:47.542387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:47:23.639743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:00.744812+0000","last_change":"2023-07-14T17:42:41.916406+0000","last_active":"2023-07-14T17:43:00.744812+0000","last_peered":"2023-07-14T17:43:00.744812+0000","last_clean":"2023-07-14T17:43:00.744812+0000","last_became_active":"2023-07-14T17:42:41.916115+0000","last_became_peered":"2023-07-14T17:42:41.916115+0000","last_unstale":"2023-07-14T17:43:00.744812+0000","last_undegraded":"2023-07-14T17:43:00.744812+0000","last_fullsized":"2023-07-14T17:43:00.744812+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:59:35.868770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835280+0000","last_change":"2023-07-14T17:42:42.529969+0000","last_active":"2023-07-14T17:42:47.835280+0000","last_peered":"2023-07-14T17:42:47.835280+0000","last_clean":"2023-07-14T17:42:47.835280+0000","last_became_active":"2023-07-14T17:42:42.529659+0000","last_became_peered":"2023-07-14T17:42:42.529659+0000","last_unstale":"2023-07-14T17:42:47.835280+0000","last_undegraded":"2023-07-14T17:42:47.835280+0000","last_fullsized":"2023-07-14T17:42:47.835280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:45:58.093667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833742+0000","last_change":"2023-07-14T17:42:46.546303+0000","last_active":"2023-07-14T17:42:47.833742+0000","last_peered":"2023-07-14T17:42:47.833742+0000","last_clean":"2023-07-14T17:42:47.833742+0000","last_became_active":"2023-07-14T17:42:46.545947+0000","last_became_peered":"2023-07-14T17:42:46.545947+0000","last_unstale":"2023-07-14T17:42:47.833742+0000","last_undegraded":"2023-07-14T17:42:47.833742+0000","last_fullsized":"2023-07-14T17:42:47.833742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:08:49.113221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.772655+0000","last_change":"2023-07-14T17:42:41.916373+0000","last_active":"2023-07-14T17:43:03.772655+0000","last_peered":"2023-07-14T17:43:03.772655+0000","last_clean":"2023-07-14T17:43:03.772655+0000","last_became_active":"2023-07-14T17:42:41.915966+0000","last_became_peered":"2023-07-14T17:42:41.915966+0000","last_unstale":"2023-07-14T17:43:03.772655+0000","last_undegraded":"2023-07-14T17:43:03.772655+0000","last_fullsized":"2023-07-14T17:43:03.772655+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:40:26.456139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059423+0000","last_change":"2023-07-14T17:42:42.532040+0000","last_active":"2023-07-14T17:42:48.059423+0000","last_peered":"2023-07-14T17:42:48.059423+0000","last_clean":"2023-07-14T17:42:48.059423+0000","last_became_active":"2023-07-14T17:42:42.531690+0000","last_became_peered":"2023-07-14T17:42:42.531690+0000","last_unstale":"2023-07-14T17:42:48.059423+0000","last_undegraded":"2023-07-14T17:42:48.059423+0000","last_fullsized":"2023-07-14T17:42:48.059423+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:36:37.658897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.319751+0000","last_change":"2023-07-14T17:42:46.550166+0000","last_active":"2023-07-14T17:42:48.319751+0000","last_peered":"2023-07-14T17:42:48.319751+0000","last_clean":"2023-07-14T17:42:48.319751+0000","last_became_active":"2023-07-14T17:42:46.549700+0000","last_became_peered":"2023-07-14T17:42:46.549700+0000","last_unstale":"2023-07-14T17:42:48.319751+0000","last_undegraded":"2023-07-14T17:42:48.319751+0000","last_fullsized":"2023-07-14T17:42:48.319751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:23.561271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.739436+0000","last_change":"2023-07-14T17:42:41.917309+0000","last_active":"2023-07-14T17:43:03.739436+0000","last_peered":"2023-07-14T17:43:03.739436+0000","last_clean":"2023-07-14T17:43:03.739436+0000","last_became_active":"2023-07-14T17:42:41.916694+0000","last_became_peered":"2023-07-14T17:42:41.916694+0000","last_unstale":"2023-07-14T17:43:03.739436+0000","last_undegraded":"2023-07-14T17:43:03.739436+0000","last_fullsized":"2023-07-14T17:43:03.739436+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:12:41.209743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539969+0000","last_change":"2023-07-14T17:42:42.531789+0000","last_active":"2023-07-14T17:42:47.539969+0000","last_peered":"2023-07-14T17:42:47.539969+0000","last_clean":"2023-07-14T17:42:47.539969+0000","last_became_active":"2023-07-14T17:42:42.531460+0000","last_became_peered":"2023-07-14T17:42:42.531460+0000","last_unstale":"2023-07-14T17:42:47.539969+0000","last_undegraded":"2023-07-14T17:42:47.539969+0000","last_fullsized":"2023-07-14T17:42:47.539969+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:17.645748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.541998+0000","last_change":"2023-07-14T17:42:46.577034+0000","last_active":"2023-07-14T17:42:47.541998+0000","last_peered":"2023-07-14T17:42:47.541998+0000","last_clean":"2023-07-14T17:42:47.541998+0000","last_became_active":"2023-07-14T17:42:46.576636+0000","last_became_peered":"2023-07-14T17:42:46.576636+0000","last_unstale":"2023-07-14T17:42:47.541998+0000","last_undegraded":"2023-07-14T17:42:47.541998+0000","last_fullsized":"2023-07-14T17:42:47.541998+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:08:40.397509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:00.743345+0000","last_change":"2023-07-14T17:42:41.919099+0000","last_active":"2023-07-14T17:43:00.743345+0000","last_peered":"2023-07-14T17:43:00.743345+0000","last_clean":"2023-07-14T17:43:00.743345+0000","last_became_active":"2023-07-14T17:42:41.918306+0000","last_became_peered":"2023-07-14T17:42:41.918306+0000","last_unstale":"2023-07-14T17:43:00.743345+0000","last_undegraded":"2023-07-14T17:43:00.743345+0000","last_fullsized":"2023-07-14T17:43:00.743345+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:20:09.398489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314097+0000","last_change":"2023-07-14T17:42:42.552969+0000","last_active":"2023-07-14T17:42:48.314097+0000","last_peered":"2023-07-14T17:42:48.314097+0000","last_clean":"2023-07-14T17:42:48.314097+0000","last_became_active":"2023-07-14T17:42:42.552647+0000","last_became_peered":"2023-07-14T17:42:42.552647+0000","last_unstale":"2023-07-14T17:42:48.314097+0000","last_undegraded":"2023-07-14T17:42:48.314097+0000","last_fullsized":"2023-07-14T17:42:48.314097+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:24:37.409124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314884+0000","last_change":"2023-07-14T17:42:46.538335+0000","last_active":"2023-07-14T17:42:48.314884+0000","last_peered":"2023-07-14T17:42:48.314884+0000","last_clean":"2023-07-14T17:42:48.314884+0000","last_became_active":"2023-07-14T17:42:46.537929+0000","last_became_peered":"2023-07-14T17:42:46.537929+0000","last_unstale":"2023-07-14T17:42:48.314884+0000","last_undegraded":"2023-07-14T17:42:48.314884+0000","last_fullsized":"2023-07-14T17:42:48.314884+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:42:43.435298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":63,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:38.863183+0000","last_change":"2023-07-14T17:42:41.916323+0000","last_active":"2023-07-14T17:43:38.863183+0000","last_peered":"2023-07-14T17:43:38.863183+0000","last_clean":"2023-07-14T17:43:38.863183+0000","last_became_active":"2023-07-14T17:42:41.916019+0000","last_became_peered":"2023-07-14T17:42:41.916019+0000","last_unstale":"2023-07-14T17:43:38.863183+0000","last_undegraded":"2023-07-14T17:43:38.863183+0000","last_fullsized":"2023-07-14T17:43:38.863183+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:42:06.067793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":38,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314443+0000","last_change":"2023-07-14T17:42:42.549843+0000","last_active":"2023-07-14T17:42:48.314443+0000","last_peered":"2023-07-14T17:42:48.314443+0000","last_clean":"2023-07-14T17:42:48.314443+0000","last_became_active":"2023-07-14T17:42:42.549347+0000","last_became_peered":"2023-07-14T17:42:42.549347+0000","last_unstale":"2023-07-14T17:42:48.314443+0000","last_undegraded":"2023-07-14T17:42:48.314443+0000","last_fullsized":"2023-07-14T17:42:48.314443+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:37:33.281681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313679+0000","last_change":"2023-07-14T17:42:46.545262+0000","last_active":"2023-07-14T17:42:48.313679+0000","last_peered":"2023-07-14T17:42:48.313679+0000","last_clean":"2023-07-14T17:42:48.313679+0000","last_became_active":"2023-07-14T17:42:46.544891+0000","last_became_peered":"2023-07-14T17:42:46.544891+0000","last_unstale":"2023-07-14T17:42:48.313679+0000","last_undegraded":"2023-07-14T17:42:48.313679+0000","last_fullsized":"2023-07-14T17:42:48.313679+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:58:10.135808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.770619+0000","last_change":"2023-07-14T17:42:41.912855+0000","last_active":"2023-07-14T17:43:03.770619+0000","last_peered":"2023-07-14T17:43:03.770619+0000","last_clean":"2023-07-14T17:43:03.770619+0000","last_became_active":"2023-07-14T17:42:41.912510+0000","last_became_peered":"2023-07-14T17:42:41.912510+0000","last_unstale":"2023-07-14T17:43:03.770619+0000","last_undegraded":"2023-07-14T17:43:03.770619+0000","last_fullsized":"2023-07-14T17:43:03.770619+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:58:29.251068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313890+0000","last_change":"2023-07-14T17:42:42.545344+0000","last_active":"2023-07-14T17:42:48.313890+0000","last_peered":"2023-07-14T17:42:48.313890+0000","last_clean":"2023-07-14T17:42:48.313890+0000","last_became_active":"2023-07-14T17:42:42.545056+0000","last_became_peered":"2023-07-14T17:42:42.545056+0000","last_unstale":"2023-07-14T17:42:48.313890+0000","last_undegraded":"2023-07-14T17:42:48.313890+0000","last_fullsized":"2023-07-14T17:42:48.313890+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:36:25.864767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539658+0000","last_change":"2023-07-14T17:42:46.541279+0000","last_active":"2023-07-14T17:42:47.539658+0000","last_peered":"2023-07-14T17:42:47.539658+0000","last_clean":"2023-07-14T17:42:47.539658+0000","last_became_active":"2023-07-14T17:42:46.540835+0000","last_became_peered":"2023-07-14T17:42:46.540835+0000","last_unstale":"2023-07-14T17:42:47.539658+0000","last_undegraded":"2023-07-14T17:42:47.539658+0000","last_fullsized":"2023-07-14T17:42:47.539658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:04:21.200711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.771177+0000","last_change":"2023-07-14T17:42:41.919257+0000","last_active":"2023-07-14T17:43:03.771177+0000","last_peered":"2023-07-14T17:43:03.771177+0000","last_clean":"2023-07-14T17:43:03.771177+0000","last_became_active":"2023-07-14T17:42:41.918968+0000","last_became_peered":"2023-07-14T17:42:41.918968+0000","last_unstale":"2023-07-14T17:43:03.771177+0000","last_undegraded":"2023-07-14T17:43:03.771177+0000","last_fullsized":"2023-07-14T17:43:03.771177+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:16:49.994602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313885+0000","last_change":"2023-07-14T17:42:42.543993+0000","last_active":"2023-07-14T17:42:48.313885+0000","last_peered":"2023-07-14T17:42:48.313885+0000","last_clean":"2023-07-14T17:42:48.313885+0000","last_became_active":"2023-07-14T17:42:42.543685+0000","last_became_peered":"2023-07-14T17:42:42.543685+0000","last_unstale":"2023-07-14T17:42:48.313885+0000","last_undegraded":"2023-07-14T17:42:48.313885+0000","last_fullsized":"2023-07-14T17:42:48.313885+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:17:07.258942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059529+0000","last_change":"2023-07-14T17:42:42.538553+0000","last_active":"2023-07-14T17:42:48.059529+0000","last_peered":"2023-07-14T17:42:48.059529+0000","last_clean":"2023-07-14T17:42:48.059529+0000","last_became_active":"2023-07-14T17:42:42.538251+0000","last_became_peered":"2023-07-14T17:42:42.538251+0000","last_unstale":"2023-07-14T17:42:48.059529+0000","last_undegraded":"2023-07-14T17:42:48.059529+0000","last_fullsized":"2023-07-14T17:42:48.059529+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:59:19.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123275+0000","last_change":"2023-07-14T17:42:42.543819+0000","last_active":"2023-07-14T17:42:48.123275+0000","last_peered":"2023-07-14T17:42:48.123275+0000","last_clean":"2023-07-14T17:42:48.123275+0000","last_became_active":"2023-07-14T17:42:42.543311+0000","last_became_peered":"2023-07-14T17:42:42.543311+0000","last_unstale":"2023-07-14T17:42:48.123275+0000","last_undegraded":"2023-07-14T17:42:48.123275+0000","last_fullsized":"2023-07-14T17:42:48.123275+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:43:18.425305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123144+0000","last_change":"2023-07-14T17:42:42.537525+0000","last_active":"2023-07-14T17:42:48.123144+0000","last_peered":"2023-07-14T17:42:48.123144+0000","last_clean":"2023-07-14T17:42:48.123144+0000","last_became_active":"2023-07-14T17:42:42.537242+0000","last_became_peered":"2023-07-14T17:42:42.537242+0000","last_unstale":"2023-07-14T17:42:48.123144+0000","last_undegraded":"2023-07-14T17:42:48.123144+0000","last_fullsized":"2023-07-14T17:42:48.123144+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:44:07.205354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313834+0000","last_change":"2023-07-14T17:42:42.547670+0000","last_active":"2023-07-14T17:42:48.313834+0000","last_peered":"2023-07-14T17:42:48.313834+0000","last_clean":"2023-07-14T17:42:48.313834+0000","last_became_active":"2023-07-14T17:42:42.547349+0000","last_became_peered":"2023-07-14T17:42:42.547349+0000","last_unstale":"2023-07-14T17:42:48.313834+0000","last_undegraded":"2023-07-14T17:42:48.313834+0000","last_fullsized":"2023-07-14T17:42:48.313834+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:11:40.630882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313964+0000","last_change":"2023-07-14T17:42:42.542741+0000","last_active":"2023-07-14T17:42:48.313964+0000","last_peered":"2023-07-14T17:42:48.313964+0000","last_clean":"2023-07-14T17:42:48.313964+0000","last_became_active":"2023-07-14T17:42:42.541972+0000","last_became_peered":"2023-07-14T17:42:42.541972+0000","last_unstale":"2023-07-14T17:42:48.313964+0000","last_undegraded":"2023-07-14T17:42:48.313964+0000","last_fullsized":"2023-07-14T17:42:48.313964+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:56:21.183121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059903+0000","last_change":"2023-07-14T17:42:42.535043+0000","last_active":"2023-07-14T17:42:48.059903+0000","last_peered":"2023-07-14T17:42:48.059903+0000","last_clean":"2023-07-14T17:42:48.059903+0000","last_became_active":"2023-07-14T17:42:42.534281+0000","last_became_peered":"2023-07-14T17:42:42.534281+0000","last_unstale":"2023-07-14T17:42:48.059903+0000","last_undegraded":"2023-07-14T17:42:48.059903+0000","last_fullsized":"2023-07-14T17:42:48.059903+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:59:12.865751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.123808+0000","last_change":"2023-07-14T17:42:42.540866+0000","last_active":"2023-07-14T17:42:48.123808+0000","last_peered":"2023-07-14T17:42:48.123808+0000","last_clean":"2023-07-14T17:42:48.123808+0000","last_became_active":"2023-07-14T17:42:42.540568+0000","last_became_peered":"2023-07-14T17:42:42.540568+0000","last_unstale":"2023-07-14T17:42:48.123808+0000","last_undegraded":"2023-07-14T17:42:48.123808+0000","last_fullsized":"2023-07-14T17:42:48.123808+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:48:50.914640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540078+0000","last_change":"2023-07-14T17:42:42.541646+0000","last_active":"2023-07-14T17:42:47.540078+0000","last_peered":"2023-07-14T17:42:47.540078+0000","last_clean":"2023-07-14T17:42:47.540078+0000","last_became_active":"2023-07-14T17:42:42.541324+0000","last_became_peered":"2023-07-14T17:42:42.541324+0000","last_unstale":"2023-07-14T17:42:47.540078+0000","last_undegraded":"2023-07-14T17:42:47.540078+0000","last_fullsized":"2023-07-14T17:42:47.540078+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:11:55.891005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539881+0000","last_change":"2023-07-14T17:42:42.534650+0000","last_active":"2023-07-14T17:42:47.539881+0000","last_peered":"2023-07-14T17:42:47.539881+0000","last_clean":"2023-07-14T17:42:47.539881+0000","last_became_active":"2023-07-14T17:42:42.534351+0000","last_became_peered":"2023-07-14T17:42:42.534351+0000","last_unstale":"2023-07-14T17:42:47.539881+0000","last_undegraded":"2023-07-14T17:42:47.539881+0000","last_fullsized":"2023-07-14T17:42:47.539881+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:13:54.843952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539359+0000","last_change":"2023-07-14T17:42:42.535815+0000","last_active":"2023-07-14T17:42:47.539359+0000","last_peered":"2023-07-14T17:42:47.539359+0000","last_clean":"2023-07-14T17:42:47.539359+0000","last_became_active":"2023-07-14T17:42:42.535510+0000","last_became_peered":"2023-07-14T17:42:42.535510+0000","last_unstale":"2023-07-14T17:42:47.539359+0000","last_undegraded":"2023-07-14T17:42:47.539359+0000","last_fullsized":"2023-07-14T17:42:47.539359+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:39:09.947314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.315221+0000","last_change":"2023-07-14T17:42:42.551758+0000","last_active":"2023-07-14T17:42:48.315221+0000","last_peered":"2023-07-14T17:42:48.315221+0000","last_clean":"2023-07-14T17:42:48.315221+0000","last_became_active":"2023-07-14T17:42:42.551444+0000","last_became_peered":"2023-07-14T17:42:42.551444+0000","last_unstale":"2023-07-14T17:42:48.315221+0000","last_undegraded":"2023-07-14T17:42:48.315221+0000","last_fullsized":"2023-07-14T17:42:48.315221+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:42:02.798071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313340+0000","last_change":"2023-07-14T17:42:42.546355+0000","last_active":"2023-07-14T17:42:48.313340+0000","last_peered":"2023-07-14T17:42:48.313340+0000","last_clean":"2023-07-14T17:42:48.313340+0000","last_became_active":"2023-07-14T17:42:42.546079+0000","last_became_peered":"2023-07-14T17:42:42.546079+0000","last_unstale":"2023-07-14T17:42:48.313340+0000","last_undegraded":"2023-07-14T17:42:48.313340+0000","last_fullsized":"2023-07-14T17:42:48.313340+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:42:39.486035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059704+0000","last_change":"2023-07-14T17:42:42.543586+0000","last_active":"2023-07-14T17:42:48.059704+0000","last_peered":"2023-07-14T17:42:48.059704+0000","last_clean":"2023-07-14T17:42:48.059704+0000","last_became_active":"2023-07-14T17:42:42.543259+0000","last_became_peered":"2023-07-14T17:42:42.543259+0000","last_unstale":"2023-07-14T17:42:48.059704+0000","last_undegraded":"2023-07-14T17:42:48.059704+0000","last_fullsized":"2023-07-14T17:42:48.059704+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:24:35.934043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65561,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":393,"num_read_kb":267,"num_write":234,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":196683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":206,"ondisk_log_size":206,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481815,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":467172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.628}]},{"osd":1,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.67100000000000004}]},{"osd":2,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.499},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.52100000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":4,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.76900000000000002}]},{"osd":5,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.69099999999999995}]},{"osd":6,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.747}]},{"osd":7,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.84499999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":9,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.92500000000000004}]},{"osd":10,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.82599999999999996}]}]},{"osd":10,"up_from":69,"seq":296352743451,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28632,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716520,"statfs":{"total":95995035648,"available":95965716480,"internally_reserved":0,"allocated":1654784,"data_stored":1112929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":1,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.496}]},{"osd":2,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.77700000000000002}]},{"osd":3,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.56899999999999995}]},{"osd":4,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.47399999999999998}]},{"osd":5,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.81399999999999995}]},{"osd":6,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.70199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":8,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.76700000000000002}]},{"osd":9,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.71899999999999997}]},{"osd":11,"last update":"Fri Jul 14 17:43:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.754}]}]},{"osd":9,"up_from":64,"seq":274877906975,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":471246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.41099999999999998,"15min":0.42399999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.497,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.45400000000000001,"15min":0.47199999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.55800000000000005,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.49099999999999999}]},{"osd":1,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.59099999999999997,"15min":0.60599999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.69899999999999995,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.44},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.60199999999999998,"15min":0.61599999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70999999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.47099999999999997}]},{"osd":2,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79900000000000004,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.64500000000000002,"15min":0.65700000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.73199999999999998,"5min":1.119,"15min":1.119},"last":0.63700000000000001}]},{"osd":3,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.48999999999999999,"15min":0.50600000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.52700000000000002,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.377},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.49199999999999999,"15min":0.505},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64400000000000002,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.40300000000000002}]},{"osd":4,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.628,"15min":0.64300000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.70699999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.63600000000000001,"15min":0.64800000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.68300000000000005,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.45200000000000001}]},{"osd":5,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.64600000000000002,"15min":0.65800000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.78400000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.624},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.65800000000000003,"15min":0.67100000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79600000000000004,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.66900000000000004}]},{"osd":6,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.372},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.52500000000000002}]},{"osd":7,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.70599999999999996,"15min":0.72799999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71099999999999997,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.71699999999999997,"15min":0.73799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.79400000000000004,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.48399999999999999}]},{"osd":8,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.749,"15min":0.76800000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.82699999999999996,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.74199999999999999,"15min":0.75800000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.85899999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.65900000000000003}]},{"osd":10,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.58299999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]}]},{"osd":8,"up_from":57,"seq":244813135908,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":467528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53400000000000003,"15min":0.53300000000000003},"min":{"1min":0.35499999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.69599999999999995}]},{"osd":1,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.372,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83899999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.40999999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.72099999999999997}]},{"osd":2,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45800000000000002,"15min":0.46100000000000002},"min":{"1min":0.27500000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60599999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.44},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.47199999999999998,"15min":0.47999999999999998},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56499999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.36899999999999999}]},{"osd":3,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.36299999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.56399999999999995}]},{"osd":4,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85099999999999998,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64300000000000002,"15min":0.64200000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89700000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.78000000000000003}]},{"osd":5,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57399999999999995,"15min":0.58099999999999996},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72499999999999998,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.312,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.75,"5min":1.018,"15min":1.018},"last":0.61799999999999999}]},{"osd":6,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.66600000000000004,"15min":0.66800000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68500000000000005,"15min":0.69299999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84799999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.57899999999999996}]},{"osd":7,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.51000000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92400000000000004,"5min":1.008,"15min":1.008},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71099999999999997,"15min":0.70999999999999996},"min":{"1min":0.48899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88200000000000001,"5min":1.258,"15min":1.258},"last":0.73999999999999999}]},{"osd":9,"last update":"Fri Jul 14 17:43:19 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.68500000000000005,"15min":0.67000000000000004},"min":{"1min":0.441,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.68600000000000005,"15min":0.67200000000000004},"min":{"1min":0.46899999999999997,"5min":0.44,"15min":0.44},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.66700000000000004}]},{"osd":10,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.754}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]}]},{"osd":7,"up_from":49,"seq":210453397544,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28560,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716592,"statfs":{"total":95995035648,"available":95965790208,"internally_reserved":0,"allocated":1581056,"data_stored":1064270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.503},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.59499999999999997}]},{"osd":1,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.41499999999999998,"15min":0.42299999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.57799999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.46000000000000002,"15min":0.47099999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.51800000000000002,"5min":0.746,"15min":0.746},"last":0.57999999999999996}]},{"osd":2,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.628,"15min":0.60299999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":5.1050000000000004,"5min":5.1050000000000004,"15min":5.1050000000000004},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.71999999999999997,"5min":1.048,"15min":1.048},"last":0.66700000000000004}]},{"osd":3,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.436,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.755},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.59899999999999998,"15min":0.61199999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72499999999999998,"5min":0.995,"15min":0.995},"last":0.80700000000000005}]},{"osd":4,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.49299999999999999,"15min":0.499},"min":{"1min":0.312,"5min":0.31,"15min":0.31},"max":{"1min":0.59999999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.52000000000000002,"15min":0.53200000000000003},"min":{"1min":0.28899999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.66900000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.72999999999999998}]},{"osd":5,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63400000000000001,"15min":0.63200000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64100000000000001,"15min":0.64300000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.94199999999999995,"5min":1.02,"15min":1.02},"last":0.89100000000000001}]},{"osd":6,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.64000000000000001,"15min":0.65000000000000002},"min":{"1min":0.435,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.878},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.84999999999999998}]},{"osd":8,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.45700000000000002,"5min":0.441,"15min":0.441},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.71299999999999997,"15min":0.72099999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0620000000000001,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.90600000000000003}]},{"osd":9,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.69399999999999995,"15min":0.70199999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.01,"5min":1.196,"15min":1.196},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.91200000000000003,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.86599999999999999}]},{"osd":10,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.83199999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]}]},{"osd":6,"up_from":44,"seq":188978561068,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":462606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.36799999999999999,"15min":0.35499999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.28199999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.308}]},{"osd":1,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57999999999999996,"15min":0.56799999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.76000000000000001,"5min":0.87,"15min":0.87},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.65000000000000002,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46300000000000002}]},{"osd":2,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59399999999999997,"15min":0.58999999999999997},"min":{"1min":0.44500000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.65500000000000003}]},{"osd":3,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.44400000000000001,"15min":0.44700000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68500000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.47199999999999998,"15min":0.45400000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78900000000000003,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.32800000000000001}]},{"osd":4,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85299999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.60799999999999998,"15min":0.60599999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81799999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.68999999999999995}]},{"osd":5,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.60099999999999998,"15min":0.59299999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.68600000000000005,"15min":0.66300000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.8679999999999999,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.60099999999999998}]},{"osd":7,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.66600000000000004,"15min":0.66000000000000003},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.68999999999999995,"15min":0.68799999999999994},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.55200000000000005}]},{"osd":8,"last update":"Fri Jul 14 17:43:04 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.73599999999999999,"15min":0.755},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.98999999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.69499999999999995,"15min":0.71099999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92500000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.64500000000000002}]},{"osd":9,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.68100000000000005,"15min":0.70099999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.91500000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.65700000000000003,"15min":0.67700000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.90200000000000002,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.48599999999999999}]},{"osd":10,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.621}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":522187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.503,"15min":0.46700000000000003},"min":{"1min":0.33800000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.497,"5min":0.51500000000000001,"15min":0.47199999999999998},"min":{"1min":0.33600000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.78600000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.70099999999999996}]},{"osd":2,"last update":"Fri Jul 14 17:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.54200000000000004,"15min":0.497},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.86099999999999999,"5min":0.879,"15min":0.879},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.52300000000000002,"15min":0.48899999999999999},"min":{"1min":0.36499999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.68300000000000005}]},{"osd":3,"last update":"Fri Jul 14 17:43:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53400000000000003,"15min":0.51600000000000001},"min":{"1min":0.31,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.75600000000000001,"5min":0.873,"15min":0.873},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57699999999999996,"15min":0.54500000000000004},"min":{"1min":0.44700000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.752,"5min":0.873,"15min":0.873},"last":0.746}]},{"osd":4,"last update":"Fri Jul 14 17:42:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.42799999999999999,"15min":0.42199999999999999},"min":{"1min":0.316,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66300000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.44,"15min":0.42799999999999999},"min":{"1min":0.29999999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.48099999999999998}]},{"osd":5,"last update":"Fri Jul 14 17:42:59 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.60899999999999999,"15min":0.58499999999999996},"min":{"1min":0.48299999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.58199999999999996,"15min":0.57099999999999995},"min":{"1min":0.42699999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.623}]},{"osd":6,"last update":"Fri Jul 14 17:43:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.41599999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.72899999999999998}]},{"osd":7,"last update":"Fri Jul 14 17:42:39 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.53000000000000003,"15min":0.51500000000000001},"min":{"1min":0.378,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.86499999999999999,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52700000000000002,"15min":0.52400000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.59199999999999997}]},{"osd":8,"last update":"Fri Jul 14 17:42:59 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.65900000000000003,"15min":0.65600000000000003},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1,"5min":1,"15min":1},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.746,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.54000000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.77200000000000002}]},{"osd":9,"last update":"Fri Jul 14 17:43:19 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65500000000000003,"15min":0.65700000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.88400000000000001,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70199999999999996,"15min":0.70099999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.83099999999999996}]},{"osd":10,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.81000000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78600000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":464062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 17:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.61499999999999999,"15min":0.53700000000000003},"min":{"1min":0.38500000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":8.5199999999999996,"5min":8.5199999999999996,"15min":8.5199999999999996},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.0960000000000001,"5min":0.65800000000000003,"15min":0.56000000000000005},"min":{"1min":0.41599999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":10.31,"5min":10.31,"15min":10.31},"last":0.62}]},{"osd":2,"last update":"Fri Jul 14 17:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55900000000000005,"15min":0.53400000000000003},"min":{"1min":0.40999999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56999999999999995,"15min":0.55300000000000005},"min":{"1min":0.379,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.53600000000000003}]},{"osd":3,"last update":"Fri Jul 14 17:41:15 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44700000000000001,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.40999999999999998,"15min":0.40600000000000003},"min":{"1min":0.27900000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.61699999999999999,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.41199999999999998}]},{"osd":4,"last update":"Fri Jul 14 17:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55700000000000005,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":1.121,"5min":0.68100000000000005,"15min":0.58099999999999996},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":10.374000000000001,"5min":10.374000000000001,"15min":10.374000000000001},"last":0.54900000000000004}]},{"osd":5,"last update":"Fri Jul 14 17:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56499999999999995,"15min":0.55100000000000005},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.755,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59599999999999997,"15min":0.58999999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84099999999999997,"5min":0.88,"15min":0.88},"last":0.58499999999999996}]},{"osd":6,"last update":"Fri Jul 14 17:43:18 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.57999999999999996,"15min":0.502},"min":{"1min":0.30599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":10.468999999999999,"5min":10.468999999999999,"15min":10.468999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.58499999999999996,"15min":0.51600000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":8.4350000000000005,"5min":8.4350000000000005,"15min":8.4350000000000005},"last":0.36699999999999999}]},{"osd":7,"last update":"Fri Jul 14 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.438}]},{"osd":8,"last update":"Fri Jul 14 17:43:06 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.68500000000000005,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.82799999999999996,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.67100000000000004,"15min":0.67900000000000005},"min":{"1min":0.46200000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.89900000000000002,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.68700000000000006}]},{"osd":9,"last update":"Fri Jul 14 17:43:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.77600000000000002,"15min":0.72799999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":10.343,"5min":10.343,"15min":10.343},"last":0.59199999999999997},{"interface":"front","average":{"1min":1.5509999999999999,"5min":0.89000000000000001,"15min":0.78000000000000003},"min":{"1min":0.45300000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":18.739000000000001,"5min":18.739000000000001,"15min":18.739000000000001},"last":0.45400000000000001}]},{"osd":10,"last update":"Fri Jul 14 17:42:44 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.73199999999999998}]},{"osd":11,"last update":"Fri Jul 14 17:43:06 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.71699999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":522476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.47499999999999998},"min":{"1min":0.41099999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.48299999999999998,"15min":0.46700000000000003},"min":{"1min":0.39100000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.72099999999999997}]},{"osd":1,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.53100000000000003,"15min":0.48299999999999998},"min":{"1min":0.35199999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.55300000000000005,"15min":0.50900000000000001},"min":{"1min":0.5,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.751}]},{"osd":3,"last update":"Fri Jul 14 17:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.57499999999999996,"15min":0.54700000000000004},"min":{"1min":0.38100000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55900000000000005,"15min":0.53900000000000003},"min":{"1min":0.40100000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.86899999999999999,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61799999999999999}]},{"osd":4,"last update":"Fri Jul 14 17:41:35 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.51900000000000002,"15min":0.51300000000000001},"min":{"1min":0.38400000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.55400000000000005,"15min":0.54600000000000004},"min":{"1min":0.39900000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.76900000000000002}]},{"osd":5,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.45600000000000002,"15min":0.43099999999999999},"min":{"1min":0.30399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.45200000000000001,"15min":0.437},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.64800000000000002}]},{"osd":6,"last update":"Fri Jul 14 17:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.60099999999999998,"15min":0.58099999999999996},"min":{"1min":0.47199999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.76000000000000001,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59899999999999998,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81799999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.81100000000000005}]},{"osd":7,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.59099999999999997,"15min":0.57499999999999996},"min":{"1min":0.40200000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.66000000000000003,"15min":0.64600000000000002},"min":{"1min":0.52500000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.85199999999999998}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.61299999999999999,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.59299999999999997,"15min":0.57399999999999995},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.92900000000000005}]},{"osd":9,"last update":"Fri Jul 14 17:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71399999999999997,"15min":0.70999999999999996},"min":{"1min":0.56999999999999995,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.76700000000000002,"15min":0.75600000000000001},"min":{"1min":0.64100000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":1.0009999999999999}]},{"osd":10,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.876},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.98799999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97499999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":462362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:16 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34300000000000003,"15min":0.32300000000000001},"min":{"1min":0.26900000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.437,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.36099999999999999,"15min":0.33200000000000002},"min":{"1min":0.28100000000000003,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.312}]},{"osd":1,"last update":"Fri Jul 14 17:43:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54200000000000004,"15min":0.50600000000000001},"min":{"1min":0.38600000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.82599999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54200000000000004,"15min":0.505},"min":{"1min":0.33300000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72499999999999998,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.496}]},{"osd":2,"last update":"Fri Jul 14 17:43:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54700000000000004,"15min":0.52400000000000002},"min":{"1min":0.41899999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.69699999999999995,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.52800000000000002},"min":{"1min":0.40899999999999997,"5min":0.379,"15min":0.379},"max":{"1min":0.71999999999999997,"5min":1.423,"15min":1.423},"last":0.42099999999999999}]},{"osd":4,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58099999999999996,"15min":0.56699999999999995},"min":{"1min":0.37,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71299999999999997,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60299999999999998,"15min":0.58199999999999996},"min":{"1min":0.50600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84199999999999997,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.67300000000000004}]},{"osd":5,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.38100000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85299999999999998,"5min":1.593,"15min":1.593},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.45400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.85699999999999998,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.54200000000000004}]},{"osd":6,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.52600000000000002,"15min":0.52900000000000003},"min":{"1min":0.32800000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.73699999999999999,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.46899999999999997,"15min":0.47599999999999998},"min":{"1min":0.32700000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.57399999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.40600000000000003}]},{"osd":7,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.63700000000000001,"15min":0.64300000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79300000000000004,"5min":1.28,"15min":1.28},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.65200000000000002,"15min":0.64400000000000002},"min":{"1min":0.56200000000000006,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.89000000000000001,"5min":1.03,"15min":1.03},"last":0.60299999999999998}]},{"osd":8,"last update":"Fri Jul 14 17:41:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.55800000000000005}]},{"osd":9,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.878,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.68100000000000005,"15min":0.69699999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.52300000000000002}]},{"osd":10,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.64300000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]},{"osd":4,"up_from":32,"seq":137438953524,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":467070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.35399999999999998,"5min":0.311,"15min":0.311},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.52100000000000002,"15min":0.502},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.496}]},{"osd":1,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.39700000000000002,"15min":0.375},"min":{"1min":0.30499999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.41099999999999998,"15min":0.39500000000000002},"min":{"1min":0.29799999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.67200000000000004,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.41199999999999998}]},{"osd":2,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.42199999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.59399999999999997,"15min":0.57199999999999995},"min":{"1min":0.437,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85399999999999998,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.54400000000000004}]},{"osd":3,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.57899999999999996,"15min":0.54500000000000004},"min":{"1min":0.39600000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.96599999999999997,"5min":1.498,"15min":1.498},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.55400000000000005,"15min":0.53200000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.84399999999999997,"5min":1.629,"15min":1.629},"last":0.442}]},{"osd":5,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.63100000000000001,"15min":0.61099999999999999},"min":{"1min":0.51500000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89800000000000002,"5min":1.256,"15min":1.256},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.624,"15min":0.60999999999999999},"min":{"1min":0.44900000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.71999999999999997}]},{"osd":6,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.72199999999999998,"15min":0.746},"min":{"1min":0.47699999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91800000000000004,"5min":2.7789999999999999,"15min":2.7789999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68200000000000005,"15min":0.66700000000000004},"min":{"1min":0.46600000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.95899999999999996,"5min":1.476,"15min":1.476},"last":0.73799999999999999}]},{"osd":7,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.55000000000000004,"15min":0.52300000000000002},"min":{"1min":0.39000000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.59399999999999997,"15min":0.57799999999999996},"min":{"1min":0.38100000000000001,"5min":0.377,"15min":0.377},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.5}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.65600000000000003,"15min":0.64400000000000002},"min":{"1min":0.41999999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.627},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72299999999999998,"15min":0.71399999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.755}]},{"osd":9,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.69399999999999995,"15min":0.69099999999999995},"min":{"1min":0.54000000000000004,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88100000000000001,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.80100000000000005,"15min":0.80400000000000005},"min":{"1min":0.61899999999999999,"5min":0.502,"15min":0.502},"max":{"1min":1.0049999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.68899999999999995}]},{"osd":10,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.67700000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]}]},{"osd":5,"up_from":37,"seq":158913790001,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1589248,"data_stored":1059558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47099999999999997,"15min":0.44700000000000001},"min":{"1min":0.35499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77800000000000002,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.48299999999999998,"15min":0.45700000000000002},"min":{"1min":0.35599999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65400000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.40799999999999997}]},{"osd":1,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.55100000000000005,"15min":0.497},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":5.3630000000000004,"5min":5.3630000000000004,"15min":5.3630000000000004},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51300000000000001,"15min":0.48999999999999999},"min":{"1min":0.42399999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.60399999999999998}]},{"osd":2,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.33500000000000002},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.376,"15min":0.35999999999999999},"min":{"1min":0.30499999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.48599999999999999}]},{"osd":3,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.501,"15min":0.495},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.51800000000000002,"15min":0.495},"min":{"1min":0.38700000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.747,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.622}]},{"osd":4,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.53500000000000003,"15min":0.50800000000000001},"min":{"1min":0.434,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78100000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54300000000000004,"15min":0.52000000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.58099999999999996}]},{"osd":6,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.39000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.75700000000000001,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.56699999999999995,"15min":0.54300000000000004},"min":{"1min":0.46400000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80800000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.67300000000000004}]},{"osd":7,"last update":"Fri Jul 14 17:43:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.58899999999999997,"15min":0.57399999999999995},"min":{"1min":0.47599999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.63300000000000001,"15min":0.61299999999999999},"min":{"1min":0.51200000000000001,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.74299999999999999}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.54700000000000004,"15min":0.54000000000000004},"min":{"1min":0.38100000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.754,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.372,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.876,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.55400000000000005}]},{"osd":9,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.56999999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91000000000000003,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68400000000000005,"15min":0.67600000000000005},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.64100000000000001}]},{"osd":10,"last update":"Fri Jul 14 17:42:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":5.3330000000000002,"5min":5.3330000000000002,"15min":5.3330000000000002},"last":0.81599999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":60629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T17:43:43.977 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T17:43:44.163 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2023-07-14T17:43:44.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2023-07-14T17:43:44.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2023-07-14T17:43:44.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2023-07-14T17:43:44.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2023-07-14T17:43:44.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [5, 8]}]: dispatch 2023-07-14T17:43:44.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2023-07-14T17:43:44.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2023-07-14T17:43:44.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2023-07-14T17:43:44.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:43 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2023-07-14T17:43:44.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2023-07-14T17:43:44.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2023-07-14T17:43:44.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2023-07-14T17:43:44.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2023-07-14T17:43:44.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2023-07-14T17:43:44.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [5, 8]}]: dispatch 2023-07-14T17:43:44.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2023-07-14T17:43:44.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2023-07-14T17:43:44.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2023-07-14T17:43:44.193 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:43 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2023-07-14T17:43:44.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2023-07-14T17:43:44.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2023-07-14T17:43:44.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2023-07-14T17:43:44.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2023-07-14T17:43:44.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]: dispatch 2023-07-14T17:43:44.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [5, 8]}]: dispatch 2023-07-14T17:43:44.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]: dispatch 2023-07-14T17:43:44.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2023-07-14T17:43:44.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]: dispatch 2023-07-14T17:43:44.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:43 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2023-07-14T17:43:44.429 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:44.936 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:44 smithi088 ceph-mon[127465]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:43:44.937 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:44 smithi088 ceph-mon[127465]: pgmap v286: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2023-07-14T17:43:44.937 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2023-07-14T17:43:44.937 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2023-07-14T17:43:44.937 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2023-07-14T17:43:44.937 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2023-07-14T17:43:44.937 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2023-07-14T17:43:44.938 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [5, 8]}]': finished 2023-07-14T17:43:44.938 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2023-07-14T17:43:44.938 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2023-07-14T17:43:44.938 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2023-07-14T17:43:44.938 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2023-07-14T17:43:44.938 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:44 smithi088 ceph-mon[127465]: osdmap e87: 12 total, 12 up, 12 in 2023-07-14T17:43:44.938 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:43:45.163 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:44 smithi178 ceph-mon[129368]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:43:45.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:44 smithi178 ceph-mon[129368]: pgmap v286: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2023-07-14T17:43:45.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2023-07-14T17:43:45.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2023-07-14T17:43:45.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2023-07-14T17:43:45.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2023-07-14T17:43:45.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2023-07-14T17:43:45.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [5, 8]}]': finished 2023-07-14T17:43:45.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2023-07-14T17:43:45.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2023-07-14T17:43:45.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2023-07-14T17:43:45.165 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2023-07-14T17:43:45.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:44 smithi178 ceph-mon[129368]: osdmap e87: 12 total, 12 up, 12 in 2023-07-14T17:43:45.166 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:43:45.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:44 smithi107 ceph-mon[128999]: from='client.15099 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:43:45.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:44 smithi107 ceph-mon[128999]: pgmap v286: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2023-07-14T17:43:45.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2023-07-14T17:43:45.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2023-07-14T17:43:45.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2023-07-14T17:43:45.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2023-07-14T17:43:45.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [4, 6]}]': finished 2023-07-14T17:43:45.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1e", "id": [5, 8]}]': finished 2023-07-14T17:43:45.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2c", "id": [4, 9]}]': finished 2023-07-14T17:43:45.225 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2023-07-14T17:43:45.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [9, 11]}]': finished 2023-07-14T17:43:45.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2023-07-14T17:43:45.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:44 smithi107 ceph-mon[128999]: osdmap e87: 12 total, 12 up, 12 in 2023-07-14T17:43:45.226 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:43:45.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.779+0000 7f384fb4f700 1 -- 172.21.15.88:0/4192258745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38480fee70 msgr2=0x7f38480ff240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:45.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.779+0000 7f384fb4f700 1 --2- 172.21.15.88:0/4192258745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38480fee70 0x7f38480ff240 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f38380099c0 tx=0x7f3838009cd0 comp rx=0 tx=0).stop 2023-07-14T17:43:45.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.779+0000 7f384fb4f700 1 -- 172.21.15.88:0/4192258745 shutdown_connections 2023-07-14T17:43:45.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.779+0000 7f384fb4f700 1 --2- 172.21.15.88:0/4192258745 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3848103f10 0x7f384810ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:45.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.779+0000 7f384fb4f700 1 --2- 172.21.15.88:0/4192258745 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f38480fee70 0x7f38480ff240 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:45.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.779+0000 7f384fb4f700 1 --2- 172.21.15.88:0/4192258745 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38480ff810 0x7f38481037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:45.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.779+0000 7f384fb4f700 1 -- 172.21.15.88:0/4192258745 >> 172.21.15.88:0/4192258745 conn(0x7f3848075d20 msgr2=0x7f3848076120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:45.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.779+0000 7f384fb4f700 1 -- 172.21.15.88:0/4192258745 shutdown_connections 2023-07-14T17:43:45.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.779+0000 7f384fb4f700 1 -- 172.21.15.88:0/4192258745 wait complete. 2023-07-14T17:43:45.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.780+0000 7f384fb4f700 1 Processor -- start 2023-07-14T17:43:45.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.780+0000 7f384fb4f700 1 -- start start 2023-07-14T17:43:45.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.780+0000 7f384fb4f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38480fee70 0x7f3848100880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:45.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.780+0000 7f384fb4f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38480ff810 0x7f3848100dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:45.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.780+0000 7f384fb4f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3848103f10 0x7f38481a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:45.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.780+0000 7f384fb4f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38481016b0 con 0x7f3848103f10 2023-07-14T17:43:45.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.780+0000 7f384fb4f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38481a43f0 con 0x7f38480fee70 2023-07-14T17:43:45.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.780+0000 7f384fb4f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38481a4560 con 0x7f38480ff810 2023-07-14T17:43:45.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.780+0000 7f384e0ec700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3848103f10 0x7f38481a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:45.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.780+0000 7f384d8eb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38480fee70 0x7f3848100880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:45.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.780+0000 7f384e0ec700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3848103f10 0x7f38481a3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42442/0 (socket says 172.21.15.88:42442) 2023-07-14T17:43:45.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.780+0000 7f384e0ec700 1 -- 172.21.15.88:0/212920920 learned_addr learned my addr 172.21.15.88:0/212920920 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:45.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.780+0000 7f384d0ea700 1 --2- 172.21.15.88:0/212920920 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38480ff810 0x7f3848100dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:45.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.781+0000 7f384e0ec700 1 -- 172.21.15.88:0/212920920 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38480ff810 msgr2=0x7f3848100dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:45.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.781+0000 7f384e0ec700 1 --2- 172.21.15.88:0/212920920 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38480ff810 0x7f3848100dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:45.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.781+0000 7f384e0ec700 1 -- 172.21.15.88:0/212920920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38480fee70 msgr2=0x7f3848100880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:45.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.781+0000 7f384e0ec700 1 --2- 172.21.15.88:0/212920920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38480fee70 0x7f3848100880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:45.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.781+0000 7f384e0ec700 1 -- 172.21.15.88:0/212920920 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38380095f0 con 0x7f3848103f10 2023-07-14T17:43:45.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.781+0000 7f384d8eb700 1 --2- 172.21.15.88:0/212920920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38480fee70 0x7f3848100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T17:43:45.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.781+0000 7f384e0ec700 1 --2- 172.21.15.88:0/212920920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3848103f10 0x7f38481a3eb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3840007ee0 tx=0x7f384000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:45.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.781+0000 7f383effd700 1 -- 172.21.15.88:0/212920920 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3840017070 con 0x7f3848103f10 2023-07-14T17:43:45.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.781+0000 7f383effd700 1 -- 172.21.15.88:0/212920920 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f384000f040 con 0x7f3848103f10 2023-07-14T17:43:45.767 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.781+0000 7f384fb4f700 1 -- 172.21.15.88:0/212920920 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38481a47e0 con 0x7f3848103f10 2023-07-14T17:43:45.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.781+0000 7f383effd700 1 -- 172.21.15.88:0/212920920 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3840005570 con 0x7f3848103f10 2023-07-14T17:43:45.768 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.781+0000 7f384fb4f700 1 -- 172.21.15.88:0/212920920 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38481a4c90 con 0x7f3848103f10 2023-07-14T17:43:45.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.782+0000 7f384fb4f700 1 -- 172.21.15.88:0/212920920 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f384804e340 con 0x7f3848103f10 2023-07-14T17:43:45.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.783+0000 7f383effd700 1 -- 172.21.15.88:0/212920920 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f384000db20 con 0x7f3848103f10 2023-07-14T17:43:45.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.783+0000 7f383effd700 1 --2- 172.21.15.88:0/212920920 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f38340a11e0 0x7f38340a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:45.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.784+0000 7f383effd700 1 -- 172.21.15.88:0/212920920 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f38400cc4b0 con 0x7f3848103f10 2023-07-14T17:43:45.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.784+0000 7f384d8eb700 1 --2- 172.21.15.88:0/212920920 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f38340a11e0 0x7f38340a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:45.769 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.784+0000 7f384d8eb700 1 --2- 172.21.15.88:0/212920920 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f38340a11e0 0x7f38340a3690 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f383800b580 tx=0x7f3838009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:45.770 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.789+0000 7f383effd700 1 -- 172.21.15.88:0/212920920 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3840095790 con 0x7f3848103f10 2023-07-14T17:43:45.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:45.990+0000 7f384fb4f700 1 -- 172.21.15.88:0/212920920 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3848103270 con 0x7f38340a11e0 2023-07-14T17:43:45.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:46.002+0000 7f383effd700 1 -- 172.21.15.88:0/212920920 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+424863 (secure 0 0 0) 0x7f3848103270 con 0x7f38340a11e0 2023-07-14T17:43:45.988 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:43:45.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:46.005+0000 7f384fb4f700 1 -- 172.21.15.88:0/212920920 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f38340a11e0 msgr2=0x7f38340a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:45.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:46.005+0000 7f384fb4f700 1 --2- 172.21.15.88:0/212920920 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f38340a11e0 0x7f38340a3690 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f383800b580 tx=0x7f3838009f10 comp rx=0 tx=0).stop 2023-07-14T17:43:45.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:46.005+0000 7f384fb4f700 1 -- 172.21.15.88:0/212920920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3848103f10 msgr2=0x7f38481a3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:45.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:46.005+0000 7f384fb4f700 1 --2- 172.21.15.88:0/212920920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3848103f10 0x7f38481a3eb0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3840007ee0 tx=0x7f384000d3b0 comp rx=0 tx=0).stop 2023-07-14T17:43:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:46.005+0000 7f384fb4f700 1 -- 172.21.15.88:0/212920920 shutdown_connections 2023-07-14T17:43:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:46.005+0000 7f384fb4f700 1 --2- 172.21.15.88:0/212920920 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f38340a11e0 0x7f38340a3690 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:46.005+0000 7f384fb4f700 1 --2- 172.21.15.88:0/212920920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3848103f10 0x7f38481a3eb0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:46.005+0000 7f384fb4f700 1 --2- 172.21.15.88:0/212920920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f38480fee70 0x7f3848100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:46.005+0000 7f384fb4f700 1 --2- 172.21.15.88:0/212920920 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38480ff810 0x7f3848100dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:46.005+0000 7f384fb4f700 1 -- 172.21.15.88:0/212920920 >> 172.21.15.88:0/212920920 conn(0x7f3848075d20 msgr2=0x7f3848102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:45.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:46.005+0000 7f384fb4f700 1 -- 172.21.15.88:0/212920920 shutdown_connections 2023-07-14T17:43:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:46.005+0000 7f384fb4f700 1 -- 172.21.15.88:0/212920920 wait complete. 2023-07-14T17:43:45.992 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-07-14T17:43:46.163 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:45 smithi178 ceph-mon[129368]: osdmap e88: 12 total, 12 up, 12 in 2023-07-14T17:43:46.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:45 smithi088 ceph-mon[127465]: osdmap e88: 12 total, 12 up, 12 in 2023-07-14T17:43:46.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:45 smithi107 ceph-mon[128999]: osdmap e88: 12 total, 12 up, 12 in 2023-07-14T17:43:46.398 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2023-07-14T17:43:45.887309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656175,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":564,"num_read_kb":401,"num_write":417,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":345,"ondisk_log_size":345,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337716,"kb_used_data":13504,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604108,"statfs":{"total":1151940427776,"available":1151594606592,"internally_reserved":0,"allocated":13828096,"data_stored":7600582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":95,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002362"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.060009+0000","last_change":"2023-07-14T17:42:42.540396+0000","last_active":"2023-07-14T17:42:48.060009+0000","last_peered":"2023-07-14T17:42:48.060009+0000","last_clean":"2023-07-14T17:42:48.060009+0000","last_became_active":"2023-07-14T17:42:42.539732+0000","last_became_peered":"2023-07-14T17:42:42.539732+0000","last_unstale":"2023-07-14T17:42:48.060009+0000","last_undegraded":"2023-07-14T17:42:48.060009+0000","last_fullsized":"2023-07-14T17:42:48.060009+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:15:44.925121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313254+0000","last_change":"2023-07-14T17:42:42.532153+0000","last_active":"2023-07-14T17:42:48.313254+0000","last_peered":"2023-07-14T17:42:48.313254+0000","last_clean":"2023-07-14T17:42:48.313254+0000","last_became_active":"2023-07-14T17:42:42.531745+0000","last_became_peered":"2023-07-14T17:42:42.531745+0000","last_unstale":"2023-07-14T17:42:48.313254+0000","last_undegraded":"2023-07-14T17:42:48.313254+0000","last_fullsized":"2023-07-14T17:42:48.313254+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:55.889251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313184+0000","last_change":"2023-07-14T17:42:42.549497+0000","last_active":"2023-07-14T17:42:48.313184+0000","last_peered":"2023-07-14T17:42:48.313184+0000","last_clean":"2023-07-14T17:42:48.313184+0000","last_became_active":"2023-07-14T17:42:42.549201+0000","last_became_peered":"2023-07-14T17:42:42.549201+0000","last_unstale":"2023-07-14T17:42:48.313184+0000","last_undegraded":"2023-07-14T17:42:48.313184+0000","last_fullsized":"2023-07-14T17:42:48.313184+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:20:16.806348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542180+0000","last_change":"2023-07-14T17:42:42.543460+0000","last_active":"2023-07-14T17:42:47.542180+0000","last_peered":"2023-07-14T17:42:47.542180+0000","last_clean":"2023-07-14T17:42:47.542180+0000","last_became_active":"2023-07-14T17:42:42.543127+0000","last_became_peered":"2023-07-14T17:42:42.543127+0000","last_unstale":"2023-07-14T17:42:47.542180+0000","last_undegraded":"2023-07-14T17:42:47.542180+0000","last_fullsized":"2023-07-14T17:42:47.542180+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:40:04.077779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835392+0000","last_change":"2023-07-14T17:42:42.534242+0000","last_active":"2023-07-14T17:42:47.835392+0000","last_peered":"2023-07-14T17:42:47.835392+0000","last_clean":"2023-07-14T17:42:47.835392+0000","last_became_active":"2023-07-14T17:42:42.533536+0000","last_became_peered":"2023-07-14T17:42:42.533536+0000","last_unstale":"2023-07-14T17:42:47.835392+0000","last_undegraded":"2023-07-14T17:42:47.835392+0000","last_fullsized":"2023-07-14T17:42:47.835392+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:57:39.927450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896541+0000","last_change":"2023-07-14T17:42:42.538042+0000","last_active":"2023-07-14T17:43:44.896541+0000","last_peered":"2023-07-14T17:43:44.896541+0000","last_clean":"2023-07-14T17:43:44.896541+0000","last_became_active":"2023-07-14T17:42:42.537612+0000","last_became_peered":"2023-07-14T17:42:42.537612+0000","last_unstale":"2023-07-14T17:43:44.896541+0000","last_undegraded":"2023-07-14T17:43:44.896541+0000","last_fullsized":"2023-07-14T17:43:44.896541+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:29:37.412187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313672+0000","last_change":"2023-07-14T17:42:42.545318+0000","last_active":"2023-07-14T17:42:48.313672+0000","last_peered":"2023-07-14T17:42:48.313672+0000","last_clean":"2023-07-14T17:42:48.313672+0000","last_became_active":"2023-07-14T17:42:42.544984+0000","last_became_peered":"2023-07-14T17:42:42.544984+0000","last_unstale":"2023-07-14T17:42:48.313672+0000","last_undegraded":"2023-07-14T17:42:48.313672+0000","last_fullsized":"2023-07-14T17:42:48.313672+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:21:01.756500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539255+0000","last_change":"2023-07-14T17:42:42.532609+0000","last_active":"2023-07-14T17:42:47.539255+0000","last_peered":"2023-07-14T17:42:47.539255+0000","last_clean":"2023-07-14T17:42:47.539255+0000","last_became_active":"2023-07-14T17:42:42.532270+0000","last_became_peered":"2023-07-14T17:42:42.532270+0000","last_unstale":"2023-07-14T17:42:47.539255+0000","last_undegraded":"2023-07-14T17:42:47.539255+0000","last_fullsized":"2023-07-14T17:42:47.539255+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:20:00.572052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T17:43:43.897259+0000","last_change":"2023-07-14T17:42:42.548973+0000","last_active":"2023-07-14T17:43:43.897259+0000","last_peered":"2023-07-14T17:43:43.897259+0000","last_clean":"2023-07-14T17:43:43.897259+0000","last_became_active":"2023-07-14T17:42:42.548672+0000","last_became_peered":"2023-07-14T17:42:42.548672+0000","last_unstale":"2023-07-14T17:43:43.897259+0000","last_undegraded":"2023-07-14T17:43:43.897259+0000","last_fullsized":"2023-07-14T17:43:43.897259+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:45:00.580622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896875+0000","last_change":"2023-07-14T17:42:42.540969+0000","last_active":"2023-07-14T17:43:44.896875+0000","last_peered":"2023-07-14T17:43:44.896875+0000","last_clean":"2023-07-14T17:43:44.896875+0000","last_became_active":"2023-07-14T17:42:42.540653+0000","last_became_peered":"2023-07-14T17:42:42.540653+0000","last_unstale":"2023-07-14T17:43:44.896875+0000","last_undegraded":"2023-07-14T17:43:44.896875+0000","last_fullsized":"2023-07-14T17:43:44.896875+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:07.062764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314444+0000","last_change":"2023-07-14T17:42:42.550016+0000","last_active":"2023-07-14T17:42:48.314444+0000","last_peered":"2023-07-14T17:42:48.314444+0000","last_clean":"2023-07-14T17:42:48.314444+0000","last_became_active":"2023-07-14T17:42:42.549653+0000","last_became_peered":"2023-07-14T17:42:42.549653+0000","last_unstale":"2023-07-14T17:42:48.314444+0000","last_undegraded":"2023-07-14T17:42:48.314444+0000","last_fullsized":"2023-07-14T17:42:48.314444+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:31:15.876030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542109+0000","last_change":"2023-07-14T17:42:42.552105+0000","last_active":"2023-07-14T17:42:47.542109+0000","last_peered":"2023-07-14T17:42:47.542109+0000","last_clean":"2023-07-14T17:42:47.542109+0000","last_became_active":"2023-07-14T17:42:42.550830+0000","last_became_peered":"2023-07-14T17:42:42.550830+0000","last_unstale":"2023-07-14T17:42:47.542109+0000","last_undegraded":"2023-07-14T17:42:47.542109+0000","last_fullsized":"2023-07-14T17:42:47.542109+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:47:37.896520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324361+0000","last_change":"2023-07-14T17:42:42.534963+0000","last_active":"2023-07-14T17:43:45.324361+0000","last_peered":"2023-07-14T17:43:45.324361+0000","last_clean":"2023-07-14T17:43:45.324361+0000","last_became_active":"2023-07-14T17:42:42.534546+0000","last_became_peered":"2023-07-14T17:42:42.534546+0000","last_unstale":"2023-07-14T17:43:45.324361+0000","last_undegraded":"2023-07-14T17:43:45.324361+0000","last_fullsized":"2023-07-14T17:43:45.324361+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:37:07.860214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.312996+0000","last_change":"2023-07-14T17:42:42.530073+0000","last_active":"2023-07-14T17:42:48.312996+0000","last_peered":"2023-07-14T17:42:48.312996+0000","last_clean":"2023-07-14T17:42:48.312996+0000","last_became_active":"2023-07-14T17:42:42.529767+0000","last_became_peered":"2023-07-14T17:42:42.529767+0000","last_unstale":"2023-07-14T17:42:48.312996+0000","last_undegraded":"2023-07-14T17:42:48.312996+0000","last_fullsized":"2023-07-14T17:42:48.312996+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:13:05.025890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.312868+0000","last_change":"2023-07-14T17:42:42.548168+0000","last_active":"2023-07-14T17:42:48.312868+0000","last_peered":"2023-07-14T17:42:48.312868+0000","last_clean":"2023-07-14T17:42:48.312868+0000","last_became_active":"2023-07-14T17:42:42.547363+0000","last_became_peered":"2023-07-14T17:42:42.547363+0000","last_unstale":"2023-07-14T17:42:48.312868+0000","last_undegraded":"2023-07-14T17:42:48.312868+0000","last_fullsized":"2023-07-14T17:42:48.312868+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:25:15.031613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T17:43:43.894470+0000","last_change":"2023-07-14T17:42:42.551042+0000","last_active":"2023-07-14T17:43:43.894470+0000","last_peered":"2023-07-14T17:43:43.894470+0000","last_clean":"2023-07-14T17:43:43.894470+0000","last_became_active":"2023-07-14T17:42:42.550730+0000","last_became_peered":"2023-07-14T17:42:42.550730+0000","last_unstale":"2023-07-14T17:43:43.894470+0000","last_undegraded":"2023-07-14T17:43:43.894470+0000","last_fullsized":"2023-07-14T17:43:43.894470+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:03:28.333415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059890+0000","last_change":"2023-07-14T17:42:42.540528+0000","last_active":"2023-07-14T17:42:48.059890+0000","last_peered":"2023-07-14T17:42:48.059890+0000","last_clean":"2023-07-14T17:42:48.059890+0000","last_became_active":"2023-07-14T17:42:42.540093+0000","last_became_peered":"2023-07-14T17:42:42.540093+0000","last_unstale":"2023-07-14T17:42:48.059890+0000","last_undegraded":"2023-07-14T17:42:48.059890+0000","last_fullsized":"2023-07-14T17:42:48.059890+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:03:51.839812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833464+0000","last_change":"2023-07-14T17:42:42.539512+0000","last_active":"2023-07-14T17:42:47.833464+0000","last_peered":"2023-07-14T17:42:47.833464+0000","last_clean":"2023-07-14T17:42:47.833464+0000","last_became_active":"2023-07-14T17:42:42.539248+0000","last_became_peered":"2023-07-14T17:42:42.539248+0000","last_unstale":"2023-07-14T17:42:47.833464+0000","last_undegraded":"2023-07-14T17:42:47.833464+0000","last_fullsized":"2023-07-14T17:42:47.833464+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:02:47.593908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895813+0000","last_change":"2023-07-14T17:42:42.531630+0000","last_active":"2023-07-14T17:43:44.895813+0000","last_peered":"2023-07-14T17:43:44.895813+0000","last_clean":"2023-07-14T17:43:44.895813+0000","last_became_active":"2023-07-14T17:42:42.531333+0000","last_became_peered":"2023-07-14T17:42:42.531333+0000","last_unstale":"2023-07-14T17:43:44.895813+0000","last_undegraded":"2023-07-14T17:43:44.895813+0000","last_fullsized":"2023-07-14T17:43:44.895813+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:37:35.706668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894477+0000","last_change":"2023-07-14T17:42:46.543062+0000","last_active":"2023-07-14T17:43:44.894477+0000","last_peered":"2023-07-14T17:43:44.894477+0000","last_clean":"2023-07-14T17:43:44.894477+0000","last_became_active":"2023-07-14T17:42:46.542189+0000","last_became_peered":"2023-07-14T17:42:46.542189+0000","last_unstale":"2023-07-14T17:43:44.894477+0000","last_undegraded":"2023-07-14T17:43:44.894477+0000","last_fullsized":"2023-07-14T17:43:44.894477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:57:03.881795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":75,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:41.854235+0000","last_change":"2023-07-14T17:42:41.914589+0000","last_active":"2023-07-14T17:43:41.854235+0000","last_peered":"2023-07-14T17:43:41.854235+0000","last_clean":"2023-07-14T17:43:41.854235+0000","last_became_active":"2023-07-14T17:42:41.914275+0000","last_became_peered":"2023-07-14T17:42:41.914275+0000","last_unstale":"2023-07-14T17:43:41.854235+0000","last_undegraded":"2023-07-14T17:43:41.854235+0000","last_fullsized":"2023-07-14T17:43:41.854235+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:09:10.487178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540020+0000","last_change":"2023-07-14T17:42:42.537207+0000","last_active":"2023-07-14T17:42:47.540020+0000","last_peered":"2023-07-14T17:42:47.540020+0000","last_clean":"2023-07-14T17:42:47.540020+0000","last_became_active":"2023-07-14T17:42:42.536879+0000","last_became_peered":"2023-07-14T17:42:42.536879+0000","last_unstale":"2023-07-14T17:42:47.540020+0000","last_undegraded":"2023-07-14T17:42:47.540020+0000","last_fullsized":"2023-07-14T17:42:47.540020+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:09:29.936504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.541631+0000","last_change":"2023-07-14T17:42:46.542500+0000","last_active":"2023-07-14T17:42:47.541631+0000","last_peered":"2023-07-14T17:42:47.541631+0000","last_clean":"2023-07-14T17:42:47.541631+0000","last_became_active":"2023-07-14T17:42:46.541932+0000","last_became_peered":"2023-07-14T17:42:46.541932+0000","last_unstale":"2023-07-14T17:42:47.541631+0000","last_undegraded":"2023-07-14T17:42:47.541631+0000","last_fullsized":"2023-07-14T17:42:47.541631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:07:53.233373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540068+0000","last_change":"2023-07-14T17:42:41.918169+0000","last_active":"2023-07-14T17:42:47.540068+0000","last_peered":"2023-07-14T17:42:47.540068+0000","last_clean":"2023-07-14T17:42:47.540068+0000","last_became_active":"2023-07-14T17:42:41.917124+0000","last_became_peered":"2023-07-14T17:42:41.917124+0000","last_unstale":"2023-07-14T17:42:47.540068+0000","last_undegraded":"2023-07-14T17:42:47.540068+0000","last_fullsized":"2023-07-14T17:42:47.540068+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:23:35.325653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313826+0000","last_change":"2023-07-14T17:42:42.539265+0000","last_active":"2023-07-14T17:42:48.313826+0000","last_peered":"2023-07-14T17:42:48.313826+0000","last_clean":"2023-07-14T17:42:48.313826+0000","last_became_active":"2023-07-14T17:42:42.538933+0000","last_became_peered":"2023-07-14T17:42:42.538933+0000","last_unstale":"2023-07-14T17:42:48.313826+0000","last_undegraded":"2023-07-14T17:42:48.313826+0000","last_fullsized":"2023-07-14T17:42:48.313826+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:08.277561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542003+0000","last_change":"2023-07-14T17:42:46.550194+0000","last_active":"2023-07-14T17:42:47.542003+0000","last_peered":"2023-07-14T17:42:47.542003+0000","last_clean":"2023-07-14T17:42:47.542003+0000","last_became_active":"2023-07-14T17:42:46.549693+0000","last_became_peered":"2023-07-14T17:42:46.549693+0000","last_unstale":"2023-07-14T17:42:47.542003+0000","last_undegraded":"2023-07-14T17:42:47.542003+0000","last_fullsized":"2023-07-14T17:42:47.542003+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:15.448341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":35,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.781365+0000","last_change":"2023-07-14T17:42:41.918769+0000","last_active":"2023-07-14T17:43:04.781365+0000","last_peered":"2023-07-14T17:43:04.781365+0000","last_clean":"2023-07-14T17:43:04.781365+0000","last_became_active":"2023-07-14T17:42:41.918573+0000","last_became_peered":"2023-07-14T17:42:41.918573+0000","last_unstale":"2023-07-14T17:43:04.781365+0000","last_undegraded":"2023-07-14T17:43:04.781365+0000","last_fullsized":"2023-07-14T17:43:04.781365+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:19:21.409733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T17:43:43.894585+0000","last_change":"2023-07-14T17:42:42.543846+0000","last_active":"2023-07-14T17:43:43.894585+0000","last_peered":"2023-07-14T17:43:43.894585+0000","last_clean":"2023-07-14T17:43:43.894585+0000","last_became_active":"2023-07-14T17:42:42.543528+0000","last_became_peered":"2023-07-14T17:42:42.543528+0000","last_unstale":"2023-07-14T17:43:43.894585+0000","last_undegraded":"2023-07-14T17:43:43.894585+0000","last_fullsized":"2023-07-14T17:43:43.894585+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:56:03.460899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.541773+0000","last_change":"2023-07-14T17:42:46.547799+0000","last_active":"2023-07-14T17:42:47.541773+0000","last_peered":"2023-07-14T17:42:47.541773+0000","last_clean":"2023-07-14T17:42:47.541773+0000","last_became_active":"2023-07-14T17:42:46.547381+0000","last_became_peered":"2023-07-14T17:42:46.547381+0000","last_unstale":"2023-07-14T17:42:47.541773+0000","last_undegraded":"2023-07-14T17:42:47.541773+0000","last_fullsized":"2023-07-14T17:42:47.541773+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:46:15.220406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.771498+0000","last_change":"2023-07-14T17:42:41.914911+0000","last_active":"2023-07-14T17:43:03.771498+0000","last_peered":"2023-07-14T17:43:03.771498+0000","last_clean":"2023-07-14T17:43:03.771498+0000","last_became_active":"2023-07-14T17:42:41.914607+0000","last_became_peered":"2023-07-14T17:42:41.914607+0000","last_unstale":"2023-07-14T17:43:03.771498+0000","last_undegraded":"2023-07-14T17:43:03.771498+0000","last_fullsized":"2023-07-14T17:43:03.771498+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:45.771966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897636+0000","last_change":"2023-07-14T17:42:42.548866+0000","last_active":"2023-07-14T17:43:44.897636+0000","last_peered":"2023-07-14T17:43:44.897636+0000","last_clean":"2023-07-14T17:43:44.897636+0000","last_became_active":"2023-07-14T17:42:42.548562+0000","last_became_peered":"2023-07-14T17:42:42.548562+0000","last_unstale":"2023-07-14T17:43:44.897636+0000","last_undegraded":"2023-07-14T17:43:44.897636+0000","last_fullsized":"2023-07-14T17:43:44.897636+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:41:33.093669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895655+0000","last_change":"2023-07-14T17:42:46.538879+0000","last_active":"2023-07-14T17:43:44.895655+0000","last_peered":"2023-07-14T17:43:44.895655+0000","last_clean":"2023-07-14T17:43:44.895655+0000","last_became_active":"2023-07-14T17:42:46.538476+0000","last_became_peered":"2023-07-14T17:42:46.538476+0000","last_unstale":"2023-07-14T17:43:44.895655+0000","last_undegraded":"2023-07-14T17:43:44.895655+0000","last_fullsized":"2023-07-14T17:43:44.895655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:50:18.995823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":75,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897652+0000","last_change":"2023-07-14T17:42:41.917215+0000","last_active":"2023-07-14T17:43:44.897652+0000","last_peered":"2023-07-14T17:43:44.897652+0000","last_clean":"2023-07-14T17:43:44.897652+0000","last_became_active":"2023-07-14T17:42:41.916640+0000","last_became_peered":"2023-07-14T17:42:41.916640+0000","last_unstale":"2023-07-14T17:43:44.897652+0000","last_undegraded":"2023-07-14T17:43:44.897652+0000","last_fullsized":"2023-07-14T17:43:44.897652+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:57:44.899889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835389+0000","last_change":"2023-07-14T17:42:42.534347+0000","last_active":"2023-07-14T17:42:47.835389+0000","last_peered":"2023-07-14T17:42:47.835389+0000","last_clean":"2023-07-14T17:42:47.835389+0000","last_became_active":"2023-07-14T17:42:42.533958+0000","last_became_peered":"2023-07-14T17:42:42.533958+0000","last_unstale":"2023-07-14T17:42:47.835389+0000","last_undegraded":"2023-07-14T17:42:47.835389+0000","last_fullsized":"2023-07-14T17:42:47.835389+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:52:39.908230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894270+0000","last_change":"2023-07-14T17:42:46.543144+0000","last_active":"2023-07-14T17:43:44.894270+0000","last_peered":"2023-07-14T17:43:44.894270+0000","last_clean":"2023-07-14T17:43:44.894270+0000","last_became_active":"2023-07-14T17:42:46.542311+0000","last_became_peered":"2023-07-14T17:42:46.542311+0000","last_unstale":"2023-07-14T17:43:44.894270+0000","last_undegraded":"2023-07-14T17:43:44.894270+0000","last_fullsized":"2023-07-14T17:43:44.894270+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:21:37.953172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896529+0000","last_change":"2023-07-14T17:42:41.916174+0000","last_active":"2023-07-14T17:43:44.896529+0000","last_peered":"2023-07-14T17:43:44.896529+0000","last_clean":"2023-07-14T17:43:44.896529+0000","last_became_active":"2023-07-14T17:42:41.915867+0000","last_became_peered":"2023-07-14T17:42:41.915867+0000","last_unstale":"2023-07-14T17:43:44.896529+0000","last_undegraded":"2023-07-14T17:43:44.896529+0000","last_fullsized":"2023-07-14T17:43:44.896529+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:59:36.014596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897789+0000","last_change":"2023-07-14T17:42:42.547064+0000","last_active":"2023-07-14T17:43:44.897789+0000","last_peered":"2023-07-14T17:43:44.897789+0000","last_clean":"2023-07-14T17:43:44.897789+0000","last_became_active":"2023-07-14T17:42:42.546777+0000","last_became_peered":"2023-07-14T17:42:42.546777+0000","last_unstale":"2023-07-14T17:43:44.897789+0000","last_undegraded":"2023-07-14T17:43:44.897789+0000","last_fullsized":"2023-07-14T17:43:44.897789+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:44:45.781755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059852+0000","last_change":"2023-07-14T17:42:46.548647+0000","last_active":"2023-07-14T17:42:48.059852+0000","last_peered":"2023-07-14T17:42:48.059852+0000","last_clean":"2023-07-14T17:42:48.059852+0000","last_became_active":"2023-07-14T17:42:46.548142+0000","last_became_peered":"2023-07-14T17:42:46.548142+0000","last_unstale":"2023-07-14T17:42:48.059852+0000","last_undegraded":"2023-07-14T17:42:48.059852+0000","last_fullsized":"2023-07-14T17:42:48.059852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:52:44.965835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324145+0000","last_change":"2023-07-14T17:42:41.911675+0000","last_active":"2023-07-14T17:43:45.324145+0000","last_peered":"2023-07-14T17:43:45.324145+0000","last_clean":"2023-07-14T17:43:45.324145+0000","last_became_active":"2023-07-14T17:42:41.911299+0000","last_became_peered":"2023-07-14T17:42:41.911299+0000","last_unstale":"2023-07-14T17:43:45.324145+0000","last_undegraded":"2023-07-14T17:43:45.324145+0000","last_fullsized":"2023-07-14T17:43:45.324145+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:37:15.112003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539414+0000","last_change":"2023-07-14T17:42:42.536211+0000","last_active":"2023-07-14T17:42:47.539414+0000","last_peered":"2023-07-14T17:42:47.539414+0000","last_clean":"2023-07-14T17:42:47.539414+0000","last_became_active":"2023-07-14T17:42:42.535763+0000","last_became_peered":"2023-07-14T17:42:42.535763+0000","last_unstale":"2023-07-14T17:42:47.539414+0000","last_undegraded":"2023-07-14T17:42:47.539414+0000","last_fullsized":"2023-07-14T17:42:47.539414+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:49:14.800525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313179+0000","last_change":"2023-07-14T17:42:46.540917+0000","last_active":"2023-07-14T17:42:48.313179+0000","last_peered":"2023-07-14T17:42:48.313179+0000","last_clean":"2023-07-14T17:42:48.313179+0000","last_became_active":"2023-07-14T17:42:46.540575+0000","last_became_peered":"2023-07-14T17:42:46.540575+0000","last_unstale":"2023-07-14T17:42:48.313179+0000","last_undegraded":"2023-07-14T17:42:48.313179+0000","last_fullsized":"2023-07-14T17:42:48.313179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:02:20.835521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897945+0000","last_change":"2023-07-14T17:42:41.917646+0000","last_active":"2023-07-14T17:43:44.897945+0000","last_peered":"2023-07-14T17:43:44.897945+0000","last_clean":"2023-07-14T17:43:44.897945+0000","last_became_active":"2023-07-14T17:42:41.917331+0000","last_became_peered":"2023-07-14T17:42:41.917331+0000","last_unstale":"2023-07-14T17:43:44.897945+0000","last_undegraded":"2023-07-14T17:43:44.897945+0000","last_fullsized":"2023-07-14T17:43:44.897945+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:52:16.441394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833094+0000","last_change":"2023-07-14T17:42:42.526696+0000","last_active":"2023-07-14T17:42:47.833094+0000","last_peered":"2023-07-14T17:42:47.833094+0000","last_clean":"2023-07-14T17:42:47.833094+0000","last_became_active":"2023-07-14T17:42:42.526353+0000","last_became_peered":"2023-07-14T17:42:42.526353+0000","last_unstale":"2023-07-14T17:42:47.833094+0000","last_undegraded":"2023-07-14T17:42:47.833094+0000","last_fullsized":"2023-07-14T17:42:47.833094+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:01:54.943614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539428+0000","last_change":"2023-07-14T17:42:46.545460+0000","last_active":"2023-07-14T17:42:47.539428+0000","last_peered":"2023-07-14T17:42:47.539428+0000","last_clean":"2023-07-14T17:42:47.539428+0000","last_became_active":"2023-07-14T17:42:46.544897+0000","last_became_peered":"2023-07-14T17:42:46.544897+0000","last_unstale":"2023-07-14T17:42:47.539428+0000","last_undegraded":"2023-07-14T17:42:47.539428+0000","last_fullsized":"2023-07-14T17:42:47.539428+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:18:28.628297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":112,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896978+0000","last_change":"2023-07-14T17:42:41.912266+0000","last_active":"2023-07-14T17:43:44.896978+0000","last_peered":"2023-07-14T17:43:44.896978+0000","last_clean":"2023-07-14T17:43:44.896978+0000","last_became_active":"2023-07-14T17:42:41.911944+0000","last_became_peered":"2023-07-14T17:42:41.911944+0000","last_unstale":"2023-07-14T17:43:44.896978+0000","last_undegraded":"2023-07-14T17:43:44.896978+0000","last_fullsized":"2023-07-14T17:43:44.896978+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:17:02.788197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":82,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T17:43:43.897418+0000","last_change":"2023-07-14T17:42:42.548609+0000","last_active":"2023-07-14T17:43:43.897418+0000","last_peered":"2023-07-14T17:43:43.897418+0000","last_clean":"2023-07-14T17:43:43.897418+0000","last_became_active":"2023-07-14T17:42:42.548253+0000","last_became_peered":"2023-07-14T17:42:42.548253+0000","last_unstale":"2023-07-14T17:43:43.897418+0000","last_undegraded":"2023-07-14T17:43:43.897418+0000","last_fullsized":"2023-07-14T17:43:43.897418+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:17:37.759289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833114+0000","last_change":"2023-07-14T17:42:46.547324+0000","last_active":"2023-07-14T17:42:47.833114+0000","last_peered":"2023-07-14T17:42:47.833114+0000","last_clean":"2023-07-14T17:42:47.833114+0000","last_became_active":"2023-07-14T17:42:46.546876+0000","last_became_peered":"2023-07-14T17:42:46.546876+0000","last_unstale":"2023-07-14T17:42:47.833114+0000","last_undegraded":"2023-07-14T17:42:47.833114+0000","last_fullsized":"2023-07-14T17:42:47.833114+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:30:13.764106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.772549+0000","last_change":"2023-07-14T17:42:41.919148+0000","last_active":"2023-07-14T17:43:03.772549+0000","last_peered":"2023-07-14T17:43:03.772549+0000","last_clean":"2023-07-14T17:43:03.772549+0000","last_became_active":"2023-07-14T17:42:41.918842+0000","last_became_peered":"2023-07-14T17:42:41.918842+0000","last_unstale":"2023-07-14T17:43:03.772549+0000","last_undegraded":"2023-07-14T17:43:03.772549+0000","last_fullsized":"2023-07-14T17:43:03.772549+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:04:34.077766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313896+0000","last_change":"2023-07-14T17:42:42.548081+0000","last_active":"2023-07-14T17:42:48.313896+0000","last_peered":"2023-07-14T17:42:48.313896+0000","last_clean":"2023-07-14T17:42:48.313896+0000","last_became_active":"2023-07-14T17:42:42.547876+0000","last_became_peered":"2023-07-14T17:42:42.547876+0000","last_unstale":"2023-07-14T17:42:48.313896+0000","last_undegraded":"2023-07-14T17:42:48.313896+0000","last_fullsized":"2023-07-14T17:42:48.313896+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:53:08.262819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542183+0000","last_change":"2023-07-14T17:42:46.550516+0000","last_active":"2023-07-14T17:42:47.542183+0000","last_peered":"2023-07-14T17:42:47.542183+0000","last_clean":"2023-07-14T17:42:47.542183+0000","last_became_active":"2023-07-14T17:42:46.549588+0000","last_became_peered":"2023-07-14T17:42:46.549588+0000","last_unstale":"2023-07-14T17:42:47.542183+0000","last_undegraded":"2023-07-14T17:42:47.542183+0000","last_fullsized":"2023-07-14T17:42:47.542183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:21.155958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897608+0000","last_change":"2023-07-14T17:42:41.917527+0000","last_active":"2023-07-14T17:43:44.897608+0000","last_peered":"2023-07-14T17:43:44.897608+0000","last_clean":"2023-07-14T17:43:44.897608+0000","last_became_active":"2023-07-14T17:42:41.917239+0000","last_became_peered":"2023-07-14T17:42:41.917239+0000","last_unstale":"2023-07-14T17:43:44.897608+0000","last_undegraded":"2023-07-14T17:43:44.897608+0000","last_fullsized":"2023-07-14T17:43:44.897608+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:07:11.390441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T17:43:43.894801+0000","last_change":"2023-07-14T17:42:42.554866+0000","last_active":"2023-07-14T17:43:43.894801+0000","last_peered":"2023-07-14T17:43:43.894801+0000","last_clean":"2023-07-14T17:43:43.894801+0000","last_became_active":"2023-07-14T17:42:42.554585+0000","last_became_peered":"2023-07-14T17:42:42.554585+0000","last_unstale":"2023-07-14T17:43:43.894801+0000","last_undegraded":"2023-07-14T17:43:43.894801+0000","last_fullsized":"2023-07-14T17:43:43.894801+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:56:27.198044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833167+0000","last_change":"2023-07-14T17:42:46.544788+0000","last_active":"2023-07-14T17:42:47.833167+0000","last_peered":"2023-07-14T17:42:47.833167+0000","last_clean":"2023-07-14T17:42:47.833167+0000","last_became_active":"2023-07-14T17:42:46.544419+0000","last_became_peered":"2023-07-14T17:42:46.544419+0000","last_unstale":"2023-07-14T17:42:47.833167+0000","last_undegraded":"2023-07-14T17:42:47.833167+0000","last_fullsized":"2023-07-14T17:42:47.833167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:03:38.940614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"86'33","reported_seq":147,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896947+0000","last_change":"2023-07-14T17:42:41.917069+0000","last_active":"2023-07-14T17:43:44.896947+0000","last_peered":"2023-07-14T17:43:44.896947+0000","last_clean":"2023-07-14T17:43:44.896947+0000","last_became_active":"2023-07-14T17:42:41.916630+0000","last_became_peered":"2023-07-14T17:42:41.916630+0000","last_unstale":"2023-07-14T17:43:44.896947+0000","last_undegraded":"2023-07-14T17:43:44.896947+0000","last_fullsized":"2023-07-14T17:43:44.896947+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:52:18.568627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58551,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":88,"num_read_kb":100,"num_write":35,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835769+0000","last_change":"2023-07-14T17:42:42.542124+0000","last_active":"2023-07-14T17:42:47.835769+0000","last_peered":"2023-07-14T17:42:47.835769+0000","last_clean":"2023-07-14T17:42:47.835769+0000","last_became_active":"2023-07-14T17:42:42.541851+0000","last_became_peered":"2023-07-14T17:42:42.541851+0000","last_unstale":"2023-07-14T17:42:47.835769+0000","last_undegraded":"2023-07-14T17:42:47.835769+0000","last_fullsized":"2023-07-14T17:42:47.835769+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:06.253283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897932+0000","last_change":"2023-07-14T17:42:46.546167+0000","last_active":"2023-07-14T17:43:44.897932+0000","last_peered":"2023-07-14T17:43:44.897932+0000","last_clean":"2023-07-14T17:43:44.897932+0000","last_became_active":"2023-07-14T17:42:46.545768+0000","last_became_peered":"2023-07-14T17:42:46.545768+0000","last_unstale":"2023-07-14T17:43:44.897932+0000","last_undegraded":"2023-07-14T17:43:44.897932+0000","last_fullsized":"2023-07-14T17:43:44.897932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:13:37.105161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894811+0000","last_change":"2023-07-14T17:42:41.918858+0000","last_active":"2023-07-14T17:43:44.894811+0000","last_peered":"2023-07-14T17:43:44.894811+0000","last_clean":"2023-07-14T17:43:44.894811+0000","last_became_active":"2023-07-14T17:42:41.918568+0000","last_became_peered":"2023-07-14T17:42:41.918568+0000","last_unstale":"2023-07-14T17:43:44.894811+0000","last_undegraded":"2023-07-14T17:43:44.894811+0000","last_fullsized":"2023-07-14T17:43:44.894811+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:43:09.338997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324334+0000","last_change":"2023-07-14T17:42:42.542310+0000","last_active":"2023-07-14T17:43:45.324334+0000","last_peered":"2023-07-14T17:43:45.324334+0000","last_clean":"2023-07-14T17:43:45.324334+0000","last_became_active":"2023-07-14T17:42:42.541989+0000","last_became_peered":"2023-07-14T17:42:42.541989+0000","last_unstale":"2023-07-14T17:43:45.324334+0000","last_undegraded":"2023-07-14T17:43:45.324334+0000","last_fullsized":"2023-07-14T17:43:45.324334+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:23:26.638546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059652+0000","last_change":"2023-07-14T17:42:46.548737+0000","last_active":"2023-07-14T17:42:48.059652+0000","last_peered":"2023-07-14T17:42:48.059652+0000","last_clean":"2023-07-14T17:42:48.059652+0000","last_became_active":"2023-07-14T17:42:46.548255+0000","last_became_peered":"2023-07-14T17:42:46.548255+0000","last_unstale":"2023-07-14T17:42:48.059652+0000","last_undegraded":"2023-07-14T17:42:48.059652+0000","last_fullsized":"2023-07-14T17:42:48.059652+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:10.311183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897638+0000","last_change":"2023-07-14T17:42:41.916382+0000","last_active":"2023-07-14T17:43:44.897638+0000","last_peered":"2023-07-14T17:43:44.897638+0000","last_clean":"2023-07-14T17:43:44.897638+0000","last_became_active":"2023-07-14T17:42:41.916082+0000","last_became_peered":"2023-07-14T17:42:41.916082+0000","last_unstale":"2023-07-14T17:43:44.897638+0000","last_undegraded":"2023-07-14T17:43:44.897638+0000","last_fullsized":"2023-07-14T17:43:44.897638+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:28:56.714038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313921+0000","last_change":"2023-07-14T17:42:42.546812+0000","last_active":"2023-07-14T17:42:48.313921+0000","last_peered":"2023-07-14T17:42:48.313921+0000","last_clean":"2023-07-14T17:42:48.313921+0000","last_became_active":"2023-07-14T17:42:42.546517+0000","last_became_peered":"2023-07-14T17:42:42.546517+0000","last_unstale":"2023-07-14T17:42:48.313921+0000","last_undegraded":"2023-07-14T17:42:48.313921+0000","last_fullsized":"2023-07-14T17:42:48.313921+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:41:22.699813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833687+0000","last_change":"2023-07-14T17:42:46.547726+0000","last_active":"2023-07-14T17:42:47.833687+0000","last_peered":"2023-07-14T17:42:47.833687+0000","last_clean":"2023-07-14T17:42:47.833687+0000","last_became_active":"2023-07-14T17:42:46.547207+0000","last_became_peered":"2023-07-14T17:42:46.547207+0000","last_unstale":"2023-07-14T17:42:47.833687+0000","last_undegraded":"2023-07-14T17:42:47.833687+0000","last_fullsized":"2023-07-14T17:42:47.833687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:55:16.418869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"86'6","reported_seq":69,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:41.850813+0000","last_change":"2023-07-14T17:42:41.913802+0000","last_active":"2023-07-14T17:43:41.850813+0000","last_peered":"2023-07-14T17:43:41.850813+0000","last_clean":"2023-07-14T17:43:41.850813+0000","last_became_active":"2023-07-14T17:42:41.913467+0000","last_became_peered":"2023-07-14T17:42:41.913467+0000","last_unstale":"2023-07-14T17:43:41.850813+0000","last_undegraded":"2023-07-14T17:43:41.850813+0000","last_fullsized":"2023-07-14T17:43:41.850813+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:46:59.725226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":42,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313504+0000","last_change":"2023-07-14T17:42:42.545379+0000","last_active":"2023-07-14T17:42:48.313504+0000","last_peered":"2023-07-14T17:42:48.313504+0000","last_clean":"2023-07-14T17:42:48.313504+0000","last_became_active":"2023-07-14T17:42:42.545012+0000","last_became_peered":"2023-07-14T17:42:42.545012+0000","last_unstale":"2023-07-14T17:42:48.313504+0000","last_undegraded":"2023-07-14T17:42:48.313504+0000","last_fullsized":"2023-07-14T17:42:48.313504+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:10.885034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324438+0000","last_change":"2023-07-14T17:42:46.544524+0000","last_active":"2023-07-14T17:43:45.324438+0000","last_peered":"2023-07-14T17:43:45.324438+0000","last_clean":"2023-07-14T17:43:45.324438+0000","last_became_active":"2023-07-14T17:42:46.544152+0000","last_became_peered":"2023-07-14T17:42:46.544152+0000","last_unstale":"2023-07-14T17:43:45.324438+0000","last_undegraded":"2023-07-14T17:43:45.324438+0000","last_fullsized":"2023-07-14T17:43:45.324438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:15:31.705985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.771510+0000","last_change":"2023-07-14T17:42:41.917282+0000","last_active":"2023-07-14T17:43:03.771510+0000","last_peered":"2023-07-14T17:43:03.771510+0000","last_clean":"2023-07-14T17:43:03.771510+0000","last_became_active":"2023-07-14T17:42:41.916994+0000","last_became_peered":"2023-07-14T17:42:41.916994+0000","last_unstale":"2023-07-14T17:43:03.771510+0000","last_undegraded":"2023-07-14T17:43:03.771510+0000","last_fullsized":"2023-07-14T17:43:03.771510+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:41:27.420237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897640+0000","last_change":"2023-07-14T17:42:42.548525+0000","last_active":"2023-07-14T17:43:44.897640+0000","last_peered":"2023-07-14T17:43:44.897640+0000","last_clean":"2023-07-14T17:43:44.897640+0000","last_became_active":"2023-07-14T17:42:42.548270+0000","last_became_peered":"2023-07-14T17:42:42.548270+0000","last_unstale":"2023-07-14T17:43:44.897640+0000","last_undegraded":"2023-07-14T17:43:44.897640+0000","last_fullsized":"2023-07-14T17:43:44.897640+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:14:31.331118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.898128+0000","last_change":"2023-07-14T17:42:46.546480+0000","last_active":"2023-07-14T17:43:44.898128+0000","last_peered":"2023-07-14T17:43:44.898128+0000","last_clean":"2023-07-14T17:43:44.898128+0000","last_became_active":"2023-07-14T17:42:46.546078+0000","last_became_peered":"2023-07-14T17:42:46.546078+0000","last_unstale":"2023-07-14T17:43:44.898128+0000","last_undegraded":"2023-07-14T17:43:44.898128+0000","last_fullsized":"2023-07-14T17:43:44.898128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:41:01.115708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.781649+0000","last_change":"2023-07-14T17:42:41.914226+0000","last_active":"2023-07-14T17:43:04.781649+0000","last_peered":"2023-07-14T17:43:04.781649+0000","last_clean":"2023-07-14T17:43:04.781649+0000","last_became_active":"2023-07-14T17:42:41.913820+0000","last_became_peered":"2023-07-14T17:42:41.913820+0000","last_unstale":"2023-07-14T17:43:04.781649+0000","last_undegraded":"2023-07-14T17:43:04.781649+0000","last_fullsized":"2023-07-14T17:43:04.781649+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:31:36.040725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059864+0000","last_change":"2023-07-14T17:42:42.543549+0000","last_active":"2023-07-14T17:42:48.059864+0000","last_peered":"2023-07-14T17:42:48.059864+0000","last_clean":"2023-07-14T17:42:48.059864+0000","last_became_active":"2023-07-14T17:42:42.543245+0000","last_became_peered":"2023-07-14T17:42:42.543245+0000","last_unstale":"2023-07-14T17:42:48.059864+0000","last_undegraded":"2023-07-14T17:42:48.059864+0000","last_fullsized":"2023-07-14T17:42:48.059864+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:30:07.889057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835441+0000","last_change":"2023-07-14T17:42:46.577245+0000","last_active":"2023-07-14T17:42:47.835441+0000","last_peered":"2023-07-14T17:42:47.835441+0000","last_clean":"2023-07-14T17:42:47.835441+0000","last_became_active":"2023-07-14T17:42:46.576793+0000","last_became_peered":"2023-07-14T17:42:46.576793+0000","last_unstale":"2023-07-14T17:42:47.835441+0000","last_undegraded":"2023-07-14T17:42:47.835441+0000","last_fullsized":"2023-07-14T17:42:47.835441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:29:32.688546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833389+0000","last_change":"2023-07-14T17:42:41.917199+0000","last_active":"2023-07-14T17:42:47.833389+0000","last_peered":"2023-07-14T17:42:47.833389+0000","last_clean":"2023-07-14T17:42:47.833389+0000","last_became_active":"2023-07-14T17:42:41.916924+0000","last_became_peered":"2023-07-14T17:42:41.916924+0000","last_unstale":"2023-07-14T17:42:47.833389+0000","last_undegraded":"2023-07-14T17:42:47.833389+0000","last_fullsized":"2023-07-14T17:42:47.833389+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:12:02.515835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896869+0000","last_change":"2023-07-14T17:42:42.540531+0000","last_active":"2023-07-14T17:43:44.896869+0000","last_peered":"2023-07-14T17:43:44.896869+0000","last_clean":"2023-07-14T17:43:44.896869+0000","last_became_active":"2023-07-14T17:42:42.540165+0000","last_became_peered":"2023-07-14T17:42:42.540165+0000","last_unstale":"2023-07-14T17:43:44.896869+0000","last_undegraded":"2023-07-14T17:43:44.896869+0000","last_fullsized":"2023-07-14T17:43:44.896869+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:30:36.053357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324213+0000","last_change":"2023-07-14T17:42:46.537158+0000","last_active":"2023-07-14T17:43:45.324213+0000","last_peered":"2023-07-14T17:43:45.324213+0000","last_clean":"2023-07-14T17:43:45.324213+0000","last_became_active":"2023-07-14T17:42:46.536816+0000","last_became_peered":"2023-07-14T17:42:46.536816+0000","last_unstale":"2023-07-14T17:43:45.324213+0000","last_undegraded":"2023-07-14T17:43:45.324213+0000","last_fullsized":"2023-07-14T17:43:45.324213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:00:28.047931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":65,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:39.852221+0000","last_change":"2023-07-14T17:42:41.913638+0000","last_active":"2023-07-14T17:43:39.852221+0000","last_peered":"2023-07-14T17:43:39.852221+0000","last_clean":"2023-07-14T17:43:39.852221+0000","last_became_active":"2023-07-14T17:42:41.913361+0000","last_became_peered":"2023-07-14T17:42:41.913361+0000","last_unstale":"2023-07-14T17:43:39.852221+0000","last_undegraded":"2023-07-14T17:43:39.852221+0000","last_fullsized":"2023-07-14T17:43:39.852221+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:30:25.777267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.901414+0000","last_change":"2023-07-14T17:43:44.901414+0000","last_active":"2023-07-14T17:43:44.901414+0000","last_peered":"2023-07-14T17:43:44.901414+0000","last_clean":"2023-07-14T17:43:44.901414+0000","last_became_active":"2023-07-14T17:43:44.901164+0000","last_became_peered":"2023-07-14T17:43:44.901164+0000","last_unstale":"2023-07-14T17:43:44.901414+0000","last_undegraded":"2023-07-14T17:43:44.901414+0000","last_fullsized":"2023-07-14T17:43:44.901414+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:15:25.526757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324220+0000","last_change":"2023-07-14T17:42:46.576999+0000","last_active":"2023-07-14T17:43:45.324220+0000","last_peered":"2023-07-14T17:43:45.324220+0000","last_clean":"2023-07-14T17:43:45.324220+0000","last_became_active":"2023-07-14T17:42:46.576652+0000","last_became_peered":"2023-07-14T17:42:46.576652+0000","last_unstale":"2023-07-14T17:43:45.324220+0000","last_undegraded":"2023-07-14T17:43:45.324220+0000","last_fullsized":"2023-07-14T17:43:45.324220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:14.124797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897667+0000","last_change":"2023-07-14T17:42:42.547053+0000","last_active":"2023-07-14T17:43:44.897667+0000","last_peered":"2023-07-14T17:43:44.897667+0000","last_clean":"2023-07-14T17:43:44.897667+0000","last_became_active":"2023-07-14T17:42:42.546735+0000","last_became_peered":"2023-07-14T17:42:42.546735+0000","last_unstale":"2023-07-14T17:43:44.897667+0000","last_undegraded":"2023-07-14T17:43:44.897667+0000","last_fullsized":"2023-07-14T17:43:44.897667+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:48:04.951463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324057+0000","last_change":"2023-07-14T17:42:41.918692+0000","last_active":"2023-07-14T17:43:45.324057+0000","last_peered":"2023-07-14T17:43:45.324057+0000","last_clean":"2023-07-14T17:43:45.324057+0000","last_became_active":"2023-07-14T17:42:41.918360+0000","last_became_peered":"2023-07-14T17:42:41.918360+0000","last_unstale":"2023-07-14T17:43:45.324057+0000","last_undegraded":"2023-07-14T17:43:45.324057+0000","last_fullsized":"2023-07-14T17:43:45.324057+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:13:06.540310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314271+0000","last_change":"2023-07-14T17:42:46.543997+0000","last_active":"2023-07-14T17:42:48.314271+0000","last_peered":"2023-07-14T17:42:48.314271+0000","last_clean":"2023-07-14T17:42:48.314271+0000","last_became_active":"2023-07-14T17:42:46.543650+0000","last_became_peered":"2023-07-14T17:42:46.543650+0000","last_unstale":"2023-07-14T17:42:48.314271+0000","last_undegraded":"2023-07-14T17:42:48.314271+0000","last_fullsized":"2023-07-14T17:42:48.314271+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:25:35.518988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539400+0000","last_change":"2023-07-14T17:42:42.548860+0000","last_active":"2023-07-14T17:42:47.539400+0000","last_peered":"2023-07-14T17:42:47.539400+0000","last_clean":"2023-07-14T17:42:47.539400+0000","last_became_active":"2023-07-14T17:42:42.548564+0000","last_became_peered":"2023-07-14T17:42:42.548564+0000","last_unstale":"2023-07-14T17:42:47.539400+0000","last_undegraded":"2023-07-14T17:42:47.539400+0000","last_fullsized":"2023-07-14T17:42:47.539400+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:07:45.387117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.782501+0000","last_change":"2023-07-14T17:42:41.912601+0000","last_active":"2023-07-14T17:43:04.782501+0000","last_peered":"2023-07-14T17:43:04.782501+0000","last_clean":"2023-07-14T17:43:04.782501+0000","last_became_active":"2023-07-14T17:42:41.912265+0000","last_became_peered":"2023-07-14T17:42:41.912265+0000","last_unstale":"2023-07-14T17:43:04.782501+0000","last_undegraded":"2023-07-14T17:43:04.782501+0000","last_fullsized":"2023-07-14T17:43:04.782501+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:58.872606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":343,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.312950+0000","last_change":"2023-07-14T17:41:43.696093+0000","last_active":"2023-07-14T17:42:48.312950+0000","last_peered":"2023-07-14T17:42:48.312950+0000","last_clean":"2023-07-14T17:42:48.312950+0000","last_became_active":"2023-07-14T17:41:43.695511+0000","last_became_peered":"2023-07-14T17:41:43.695511+0000","last_unstale":"2023-07-14T17:42:48.312950+0000","last_undegraded":"2023-07-14T17:42:48.312950+0000","last_fullsized":"2023-07-14T17:42:48.312950+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:38:50.657865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:38:50.657865+0000","last_clean_scrub_stamp":"2023-07-14T17:38:50.657865+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:30:55.242715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"activating","last_fresh":"2023-07-14T17:43:44.900723+0000","last_change":"2023-07-14T17:43:44.897633+0000","last_active":"2023-07-14T17:43:43.899017+0000","last_peered":"2023-07-14T17:42:50.079160+0000","last_clean":"2023-07-14T17:42:50.079160+0000","last_became_active":"2023-07-14T17:42:46.577067+0000","last_became_peered":"2023-07-14T17:42:46.577067+0000","last_unstale":"2023-07-14T17:43:44.900723+0000","last_undegraded":"2023-07-14T17:43:44.900723+0000","last_fullsized":"2023-07-14T17:43:44.900723+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:06:30.535779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":["5(0)","6(2)","9(1)","10(3)","11(1)"],"object_location_counts":[],"blocked_by":[2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.323905+0000","last_change":"2023-07-14T17:42:42.543243+0000","last_active":"2023-07-14T17:43:45.323905+0000","last_peered":"2023-07-14T17:43:45.323905+0000","last_clean":"2023-07-14T17:43:45.323905+0000","last_became_active":"2023-07-14T17:42:42.542965+0000","last_became_peered":"2023-07-14T17:42:42.542965+0000","last_unstale":"2023-07-14T17:43:45.323905+0000","last_undegraded":"2023-07-14T17:43:45.323905+0000","last_fullsized":"2023-07-14T17:43:45.323905+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:11:31.220902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:01.753945+0000","last_change":"2023-07-14T17:42:41.920806+0000","last_active":"2023-07-14T17:43:01.753945+0000","last_peered":"2023-07-14T17:43:01.753945+0000","last_clean":"2023-07-14T17:43:01.753945+0000","last_became_active":"2023-07-14T17:42:41.920528+0000","last_became_peered":"2023-07-14T17:42:41.920528+0000","last_unstale":"2023-07-14T17:43:01.753945+0000","last_undegraded":"2023-07-14T17:43:01.753945+0000","last_fullsized":"2023-07-14T17:43:01.753945+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:43:43.408841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059657+0000","last_change":"2023-07-14T17:42:46.542026+0000","last_active":"2023-07-14T17:42:48.059657+0000","last_peered":"2023-07-14T17:42:48.059657+0000","last_clean":"2023-07-14T17:42:48.059657+0000","last_became_active":"2023-07-14T17:42:46.541672+0000","last_became_peered":"2023-07-14T17:42:46.541672+0000","last_unstale":"2023-07-14T17:42:48.059657+0000","last_undegraded":"2023-07-14T17:42:48.059657+0000","last_fullsized":"2023-07-14T17:42:48.059657+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:45:35.131993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.772096+0000","last_change":"2023-07-14T17:42:41.915798+0000","last_active":"2023-07-14T17:43:03.772096+0000","last_peered":"2023-07-14T17:43:03.772096+0000","last_clean":"2023-07-14T17:43:03.772096+0000","last_became_active":"2023-07-14T17:42:41.915492+0000","last_became_peered":"2023-07-14T17:42:41.915492+0000","last_unstale":"2023-07-14T17:43:03.772096+0000","last_undegraded":"2023-07-14T17:43:03.772096+0000","last_fullsized":"2023-07-14T17:43:03.772096+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:25:42.031839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324401+0000","last_change":"2023-07-14T17:42:42.533500+0000","last_active":"2023-07-14T17:43:45.324401+0000","last_peered":"2023-07-14T17:43:45.324401+0000","last_clean":"2023-07-14T17:43:45.324401+0000","last_became_active":"2023-07-14T17:42:42.533232+0000","last_became_peered":"2023-07-14T17:42:42.533232+0000","last_unstale":"2023-07-14T17:43:45.324401+0000","last_undegraded":"2023-07-14T17:43:45.324401+0000","last_fullsized":"2023-07-14T17:43:45.324401+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:42:49.105557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539366+0000","last_change":"2023-07-14T17:42:46.577131+0000","last_active":"2023-07-14T17:42:47.539366+0000","last_peered":"2023-07-14T17:42:47.539366+0000","last_clean":"2023-07-14T17:42:47.539366+0000","last_became_active":"2023-07-14T17:42:46.576695+0000","last_became_peered":"2023-07-14T17:42:46.576695+0000","last_unstale":"2023-07-14T17:42:47.539366+0000","last_undegraded":"2023-07-14T17:42:47.539366+0000","last_fullsized":"2023-07-14T17:42:47.539366+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:49:05.554462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'9","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:07.833208+0000","last_change":"2023-07-14T17:42:41.919021+0000","last_active":"2023-07-14T17:43:07.833208+0000","last_peered":"2023-07-14T17:43:07.833208+0000","last_clean":"2023-07-14T17:43:07.833208+0000","last_became_active":"2023-07-14T17:42:41.918714+0000","last_became_peered":"2023-07-14T17:42:41.918714+0000","last_unstale":"2023-07-14T17:43:07.833208+0000","last_undegraded":"2023-07-14T17:43:07.833208+0000","last_fullsized":"2023-07-14T17:43:07.833208+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:40:48.400679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539836+0000","last_change":"2023-07-14T17:42:42.534661+0000","last_active":"2023-07-14T17:42:47.539836+0000","last_peered":"2023-07-14T17:42:47.539836+0000","last_clean":"2023-07-14T17:42:47.539836+0000","last_became_active":"2023-07-14T17:42:42.534357+0000","last_became_peered":"2023-07-14T17:42:42.534357+0000","last_unstale":"2023-07-14T17:42:47.539836+0000","last_undegraded":"2023-07-14T17:42:47.539836+0000","last_fullsized":"2023-07-14T17:42:47.539836+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:24:53.645011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894357+0000","last_change":"2023-07-14T17:42:46.538648+0000","last_active":"2023-07-14T17:43:44.894357+0000","last_peered":"2023-07-14T17:43:44.894357+0000","last_clean":"2023-07-14T17:43:44.894357+0000","last_became_active":"2023-07-14T17:42:46.538225+0000","last_became_peered":"2023-07-14T17:42:46.538225+0000","last_unstale":"2023-07-14T17:43:44.894357+0000","last_undegraded":"2023-07-14T17:43:44.894357+0000","last_fullsized":"2023-07-14T17:43:44.894357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:21:00.195671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.772287+0000","last_change":"2023-07-14T17:42:41.915406+0000","last_active":"2023-07-14T17:43:03.772287+0000","last_peered":"2023-07-14T17:43:03.772287+0000","last_clean":"2023-07-14T17:43:03.772287+0000","last_became_active":"2023-07-14T17:42:41.914967+0000","last_became_peered":"2023-07-14T17:42:41.914967+0000","last_unstale":"2023-07-14T17:43:03.772287+0000","last_undegraded":"2023-07-14T17:43:03.772287+0000","last_fullsized":"2023-07-14T17:43:03.772287+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:56:07.113627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313341+0000","last_change":"2023-07-14T17:42:42.539052+0000","last_active":"2023-07-14T17:42:48.313341+0000","last_peered":"2023-07-14T17:42:48.313341+0000","last_clean":"2023-07-14T17:42:48.313341+0000","last_became_active":"2023-07-14T17:42:42.538752+0000","last_became_peered":"2023-07-14T17:42:42.538752+0000","last_unstale":"2023-07-14T17:42:48.313341+0000","last_undegraded":"2023-07-14T17:42:48.313341+0000","last_fullsized":"2023-07-14T17:42:48.313341+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:25:31.544906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542387+0000","last_change":"2023-07-14T17:42:46.577224+0000","last_active":"2023-07-14T17:42:47.542387+0000","last_peered":"2023-07-14T17:42:47.542387+0000","last_clean":"2023-07-14T17:42:47.542387+0000","last_became_active":"2023-07-14T17:42:46.576781+0000","last_became_peered":"2023-07-14T17:42:46.576781+0000","last_unstale":"2023-07-14T17:42:47.542387+0000","last_undegraded":"2023-07-14T17:42:47.542387+0000","last_fullsized":"2023-07-14T17:42:47.542387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:47:23.639743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T17:43:43.894174+0000","last_change":"2023-07-14T17:42:41.916406+0000","last_active":"2023-07-14T17:43:43.894174+0000","last_peered":"2023-07-14T17:43:43.894174+0000","last_clean":"2023-07-14T17:43:43.894174+0000","last_became_active":"2023-07-14T17:42:41.916115+0000","last_became_peered":"2023-07-14T17:42:41.916115+0000","last_unstale":"2023-07-14T17:43:43.894174+0000","last_undegraded":"2023-07-14T17:43:43.894174+0000","last_fullsized":"2023-07-14T17:43:43.894174+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:24:40.609115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835280+0000","last_change":"2023-07-14T17:42:42.529969+0000","last_active":"2023-07-14T17:42:47.835280+0000","last_peered":"2023-07-14T17:42:47.835280+0000","last_clean":"2023-07-14T17:42:47.835280+0000","last_became_active":"2023-07-14T17:42:42.529659+0000","last_became_peered":"2023-07-14T17:42:42.529659+0000","last_unstale":"2023-07-14T17:42:47.835280+0000","last_undegraded":"2023-07-14T17:42:47.835280+0000","last_fullsized":"2023-07-14T17:42:47.835280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:45:58.093667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833742+0000","last_change":"2023-07-14T17:42:46.546303+0000","last_active":"2023-07-14T17:42:47.833742+0000","last_peered":"2023-07-14T17:42:47.833742+0000","last_clean":"2023-07-14T17:42:47.833742+0000","last_became_active":"2023-07-14T17:42:46.545947+0000","last_became_peered":"2023-07-14T17:42:46.545947+0000","last_unstale":"2023-07-14T17:42:47.833742+0000","last_undegraded":"2023-07-14T17:42:47.833742+0000","last_fullsized":"2023-07-14T17:42:47.833742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:08:49.113221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.772655+0000","last_change":"2023-07-14T17:42:41.916373+0000","last_active":"2023-07-14T17:43:03.772655+0000","last_peered":"2023-07-14T17:43:03.772655+0000","last_clean":"2023-07-14T17:43:03.772655+0000","last_became_active":"2023-07-14T17:42:41.915966+0000","last_became_peered":"2023-07-14T17:42:41.915966+0000","last_unstale":"2023-07-14T17:43:03.772655+0000","last_undegraded":"2023-07-14T17:43:03.772655+0000","last_fullsized":"2023-07-14T17:43:03.772655+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:40:26.456139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059423+0000","last_change":"2023-07-14T17:42:42.532040+0000","last_active":"2023-07-14T17:42:48.059423+0000","last_peered":"2023-07-14T17:42:48.059423+0000","last_clean":"2023-07-14T17:42:48.059423+0000","last_became_active":"2023-07-14T17:42:42.531690+0000","last_became_peered":"2023-07-14T17:42:42.531690+0000","last_unstale":"2023-07-14T17:42:48.059423+0000","last_undegraded":"2023-07-14T17:42:48.059423+0000","last_fullsized":"2023-07-14T17:42:48.059423+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:36:37.658897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.319751+0000","last_change":"2023-07-14T17:42:46.550166+0000","last_active":"2023-07-14T17:42:48.319751+0000","last_peered":"2023-07-14T17:42:48.319751+0000","last_clean":"2023-07-14T17:42:48.319751+0000","last_became_active":"2023-07-14T17:42:46.549700+0000","last_became_peered":"2023-07-14T17:42:46.549700+0000","last_unstale":"2023-07-14T17:42:48.319751+0000","last_undegraded":"2023-07-14T17:42:48.319751+0000","last_fullsized":"2023-07-14T17:42:48.319751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:23.561271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.739436+0000","last_change":"2023-07-14T17:42:41.917309+0000","last_active":"2023-07-14T17:43:03.739436+0000","last_peered":"2023-07-14T17:43:03.739436+0000","last_clean":"2023-07-14T17:43:03.739436+0000","last_became_active":"2023-07-14T17:42:41.916694+0000","last_became_peered":"2023-07-14T17:42:41.916694+0000","last_unstale":"2023-07-14T17:43:03.739436+0000","last_undegraded":"2023-07-14T17:43:03.739436+0000","last_fullsized":"2023-07-14T17:43:03.739436+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:12:41.209743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539969+0000","last_change":"2023-07-14T17:42:42.531789+0000","last_active":"2023-07-14T17:42:47.539969+0000","last_peered":"2023-07-14T17:42:47.539969+0000","last_clean":"2023-07-14T17:42:47.539969+0000","last_became_active":"2023-07-14T17:42:42.531460+0000","last_became_peered":"2023-07-14T17:42:42.531460+0000","last_unstale":"2023-07-14T17:42:47.539969+0000","last_undegraded":"2023-07-14T17:42:47.539969+0000","last_fullsized":"2023-07-14T17:42:47.539969+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:17.645748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.541998+0000","last_change":"2023-07-14T17:42:46.577034+0000","last_active":"2023-07-14T17:42:47.541998+0000","last_peered":"2023-07-14T17:42:47.541998+0000","last_clean":"2023-07-14T17:42:47.541998+0000","last_became_active":"2023-07-14T17:42:46.576636+0000","last_became_peered":"2023-07-14T17:42:46.576636+0000","last_unstale":"2023-07-14T17:42:47.541998+0000","last_undegraded":"2023-07-14T17:42:47.541998+0000","last_fullsized":"2023-07-14T17:42:47.541998+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:08:40.397509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:00.743345+0000","last_change":"2023-07-14T17:42:41.919099+0000","last_active":"2023-07-14T17:43:00.743345+0000","last_peered":"2023-07-14T17:43:00.743345+0000","last_clean":"2023-07-14T17:43:00.743345+0000","last_became_active":"2023-07-14T17:42:41.918306+0000","last_became_peered":"2023-07-14T17:42:41.918306+0000","last_unstale":"2023-07-14T17:43:00.743345+0000","last_undegraded":"2023-07-14T17:43:00.743345+0000","last_fullsized":"2023-07-14T17:43:00.743345+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:20:09.398489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314097+0000","last_change":"2023-07-14T17:42:42.552969+0000","last_active":"2023-07-14T17:42:48.314097+0000","last_peered":"2023-07-14T17:42:48.314097+0000","last_clean":"2023-07-14T17:42:48.314097+0000","last_became_active":"2023-07-14T17:42:42.552647+0000","last_became_peered":"2023-07-14T17:42:42.552647+0000","last_unstale":"2023-07-14T17:42:48.314097+0000","last_undegraded":"2023-07-14T17:42:48.314097+0000","last_fullsized":"2023-07-14T17:42:48.314097+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:24:37.409124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T17:43:43.894717+0000","last_change":"2023-07-14T17:42:46.538335+0000","last_active":"2023-07-14T17:43:43.894717+0000","last_peered":"2023-07-14T17:43:43.894717+0000","last_clean":"2023-07-14T17:43:43.894717+0000","last_became_active":"2023-07-14T17:42:46.537929+0000","last_became_peered":"2023-07-14T17:42:46.537929+0000","last_unstale":"2023-07-14T17:43:43.894717+0000","last_undegraded":"2023-07-14T17:43:43.894717+0000","last_fullsized":"2023-07-14T17:43:43.894717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:44:37.297601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":72,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894655+0000","last_change":"2023-07-14T17:42:41.916323+0000","last_active":"2023-07-14T17:43:44.894655+0000","last_peered":"2023-07-14T17:43:44.894655+0000","last_clean":"2023-07-14T17:43:44.894655+0000","last_became_active":"2023-07-14T17:42:41.916019+0000","last_became_peered":"2023-07-14T17:42:41.916019+0000","last_unstale":"2023-07-14T17:43:44.894655+0000","last_undegraded":"2023-07-14T17:43:44.894655+0000","last_fullsized":"2023-07-14T17:43:44.894655+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:29:56.282750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.314443+0000","last_change":"2023-07-14T17:42:42.549843+0000","last_active":"2023-07-14T17:42:48.314443+0000","last_peered":"2023-07-14T17:42:48.314443+0000","last_clean":"2023-07-14T17:42:48.314443+0000","last_became_active":"2023-07-14T17:42:42.549347+0000","last_became_peered":"2023-07-14T17:42:42.549347+0000","last_unstale":"2023-07-14T17:42:48.314443+0000","last_undegraded":"2023-07-14T17:42:48.314443+0000","last_fullsized":"2023-07-14T17:42:48.314443+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:37:33.281681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313679+0000","last_change":"2023-07-14T17:42:46.545262+0000","last_active":"2023-07-14T17:42:48.313679+0000","last_peered":"2023-07-14T17:42:48.313679+0000","last_clean":"2023-07-14T17:42:48.313679+0000","last_became_active":"2023-07-14T17:42:46.544891+0000","last_became_peered":"2023-07-14T17:42:46.544891+0000","last_unstale":"2023-07-14T17:42:48.313679+0000","last_undegraded":"2023-07-14T17:42:48.313679+0000","last_fullsized":"2023-07-14T17:42:48.313679+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:58:10.135808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897783+0000","last_change":"2023-07-14T17:42:41.912855+0000","last_active":"2023-07-14T17:43:44.897783+0000","last_peered":"2023-07-14T17:43:44.897783+0000","last_clean":"2023-07-14T17:43:44.897783+0000","last_became_active":"2023-07-14T17:42:41.912510+0000","last_became_peered":"2023-07-14T17:42:41.912510+0000","last_unstale":"2023-07-14T17:43:44.897783+0000","last_undegraded":"2023-07-14T17:43:44.897783+0000","last_fullsized":"2023-07-14T17:43:44.897783+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:29:49.003962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313890+0000","last_change":"2023-07-14T17:42:42.545344+0000","last_active":"2023-07-14T17:42:48.313890+0000","last_peered":"2023-07-14T17:42:48.313890+0000","last_clean":"2023-07-14T17:42:48.313890+0000","last_became_active":"2023-07-14T17:42:42.545056+0000","last_became_peered":"2023-07-14T17:42:42.545056+0000","last_unstale":"2023-07-14T17:42:48.313890+0000","last_undegraded":"2023-07-14T17:42:48.313890+0000","last_fullsized":"2023-07-14T17:42:48.313890+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:36:25.864767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897362+0000","last_change":"2023-07-14T17:42:46.541279+0000","last_active":"2023-07-14T17:43:44.897362+0000","last_peered":"2023-07-14T17:43:44.897362+0000","last_clean":"2023-07-14T17:43:44.897362+0000","last_became_active":"2023-07-14T17:42:46.540835+0000","last_became_peered":"2023-07-14T17:42:46.540835+0000","last_unstale":"2023-07-14T17:43:44.897362+0000","last_undegraded":"2023-07-14T17:43:44.897362+0000","last_fullsized":"2023-07-14T17:43:44.897362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:41:42.325967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324578+0000","last_change":"2023-07-14T17:42:41.919257+0000","last_active":"2023-07-14T17:43:45.324578+0000","last_peered":"2023-07-14T17:43:45.324578+0000","last_clean":"2023-07-14T17:43:45.324578+0000","last_became_active":"2023-07-14T17:42:41.918968+0000","last_became_peered":"2023-07-14T17:42:41.918968+0000","last_unstale":"2023-07-14T17:43:45.324578+0000","last_undegraded":"2023-07-14T17:43:45.324578+0000","last_fullsized":"2023-07-14T17:43:45.324578+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:06:24.227953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313885+0000","last_change":"2023-07-14T17:42:42.543993+0000","last_active":"2023-07-14T17:42:48.313885+0000","last_peered":"2023-07-14T17:42:48.313885+0000","last_clean":"2023-07-14T17:42:48.313885+0000","last_became_active":"2023-07-14T17:42:42.543685+0000","last_became_peered":"2023-07-14T17:42:42.543685+0000","last_unstale":"2023-07-14T17:42:48.313885+0000","last_undegraded":"2023-07-14T17:42:48.313885+0000","last_fullsized":"2023-07-14T17:42:48.313885+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:17:07.258942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059529+0000","last_change":"2023-07-14T17:42:42.538553+0000","last_active":"2023-07-14T17:42:48.059529+0000","last_peered":"2023-07-14T17:42:48.059529+0000","last_clean":"2023-07-14T17:42:48.059529+0000","last_became_active":"2023-07-14T17:42:42.538251+0000","last_became_peered":"2023-07-14T17:42:42.538251+0000","last_unstale":"2023-07-14T17:42:48.059529+0000","last_undegraded":"2023-07-14T17:42:48.059529+0000","last_fullsized":"2023-07-14T17:42:48.059529+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:59:19.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894621+0000","last_change":"2023-07-14T17:42:42.543819+0000","last_active":"2023-07-14T17:43:44.894621+0000","last_peered":"2023-07-14T17:43:44.894621+0000","last_clean":"2023-07-14T17:43:44.894621+0000","last_became_active":"2023-07-14T17:42:42.543311+0000","last_became_peered":"2023-07-14T17:42:42.543311+0000","last_unstale":"2023-07-14T17:43:44.894621+0000","last_undegraded":"2023-07-14T17:43:44.894621+0000","last_fullsized":"2023-07-14T17:43:44.894621+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:03:11.132113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894009+0000","last_change":"2023-07-14T17:42:42.537525+0000","last_active":"2023-07-14T17:43:44.894009+0000","last_peered":"2023-07-14T17:43:44.894009+0000","last_clean":"2023-07-14T17:43:44.894009+0000","last_became_active":"2023-07-14T17:42:42.537242+0000","last_became_peered":"2023-07-14T17:42:42.537242+0000","last_unstale":"2023-07-14T17:43:44.894009+0000","last_undegraded":"2023-07-14T17:43:44.894009+0000","last_fullsized":"2023-07-14T17:43:44.894009+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:40:58.077332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313834+0000","last_change":"2023-07-14T17:42:42.547670+0000","last_active":"2023-07-14T17:42:48.313834+0000","last_peered":"2023-07-14T17:42:48.313834+0000","last_clean":"2023-07-14T17:42:48.313834+0000","last_became_active":"2023-07-14T17:42:42.547349+0000","last_became_peered":"2023-07-14T17:42:42.547349+0000","last_unstale":"2023-07-14T17:42:48.313834+0000","last_undegraded":"2023-07-14T17:42:48.313834+0000","last_fullsized":"2023-07-14T17:42:48.313834+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:11:40.630882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324379+0000","last_change":"2023-07-14T17:42:42.542741+0000","last_active":"2023-07-14T17:43:45.324379+0000","last_peered":"2023-07-14T17:43:45.324379+0000","last_clean":"2023-07-14T17:43:45.324379+0000","last_became_active":"2023-07-14T17:42:42.541972+0000","last_became_peered":"2023-07-14T17:42:42.541972+0000","last_unstale":"2023-07-14T17:43:45.324379+0000","last_undegraded":"2023-07-14T17:43:45.324379+0000","last_fullsized":"2023-07-14T17:43:45.324379+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:27.843013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059903+0000","last_change":"2023-07-14T17:42:42.535043+0000","last_active":"2023-07-14T17:42:48.059903+0000","last_peered":"2023-07-14T17:42:48.059903+0000","last_clean":"2023-07-14T17:42:48.059903+0000","last_became_active":"2023-07-14T17:42:42.534281+0000","last_became_peered":"2023-07-14T17:42:42.534281+0000","last_unstale":"2023-07-14T17:42:48.059903+0000","last_undegraded":"2023-07-14T17:42:48.059903+0000","last_fullsized":"2023-07-14T17:42:48.059903+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:59:12.865751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895460+0000","last_change":"2023-07-14T17:42:42.540866+0000","last_active":"2023-07-14T17:43:44.895460+0000","last_peered":"2023-07-14T17:43:44.895460+0000","last_clean":"2023-07-14T17:43:44.895460+0000","last_became_active":"2023-07-14T17:42:42.540568+0000","last_became_peered":"2023-07-14T17:42:42.540568+0000","last_unstale":"2023-07-14T17:43:44.895460+0000","last_undegraded":"2023-07-14T17:43:44.895460+0000","last_fullsized":"2023-07-14T17:43:44.895460+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:12:01.698385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.540078+0000","last_change":"2023-07-14T17:42:42.541646+0000","last_active":"2023-07-14T17:42:47.540078+0000","last_peered":"2023-07-14T17:42:47.540078+0000","last_clean":"2023-07-14T17:42:47.540078+0000","last_became_active":"2023-07-14T17:42:42.541324+0000","last_became_peered":"2023-07-14T17:42:42.541324+0000","last_unstale":"2023-07-14T17:42:47.540078+0000","last_undegraded":"2023-07-14T17:42:47.540078+0000","last_fullsized":"2023-07-14T17:42:47.540078+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:11:55.891005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897528+0000","last_change":"2023-07-14T17:42:42.534650+0000","last_active":"2023-07-14T17:43:44.897528+0000","last_peered":"2023-07-14T17:43:44.897528+0000","last_clean":"2023-07-14T17:43:44.897528+0000","last_became_active":"2023-07-14T17:42:42.534351+0000","last_became_peered":"2023-07-14T17:42:42.534351+0000","last_unstale":"2023-07-14T17:43:44.897528+0000","last_undegraded":"2023-07-14T17:43:44.897528+0000","last_fullsized":"2023-07-14T17:43:44.897528+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:13:52.397350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539359+0000","last_change":"2023-07-14T17:42:42.535815+0000","last_active":"2023-07-14T17:42:47.539359+0000","last_peered":"2023-07-14T17:42:47.539359+0000","last_clean":"2023-07-14T17:42:47.539359+0000","last_became_active":"2023-07-14T17:42:42.535510+0000","last_became_peered":"2023-07-14T17:42:42.535510+0000","last_unstale":"2023-07-14T17:42:47.539359+0000","last_undegraded":"2023-07-14T17:42:47.539359+0000","last_fullsized":"2023-07-14T17:42:47.539359+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:39:09.947314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T17:43:43.895262+0000","last_change":"2023-07-14T17:42:42.551758+0000","last_active":"2023-07-14T17:43:43.895262+0000","last_peered":"2023-07-14T17:43:43.895262+0000","last_clean":"2023-07-14T17:43:43.895262+0000","last_became_active":"2023-07-14T17:42:42.551444+0000","last_became_peered":"2023-07-14T17:42:42.551444+0000","last_unstale":"2023-07-14T17:43:43.895262+0000","last_undegraded":"2023-07-14T17:43:43.895262+0000","last_fullsized":"2023-07-14T17:43:43.895262+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:52:15.412415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324115+0000","last_change":"2023-07-14T17:42:42.546355+0000","last_active":"2023-07-14T17:43:45.324115+0000","last_peered":"2023-07-14T17:43:45.324115+0000","last_clean":"2023-07-14T17:43:45.324115+0000","last_became_active":"2023-07-14T17:42:42.546079+0000","last_became_peered":"2023-07-14T17:42:42.546079+0000","last_unstale":"2023-07-14T17:43:45.324115+0000","last_undegraded":"2023-07-14T17:43:45.324115+0000","last_fullsized":"2023-07-14T17:43:45.324115+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:57:16.604025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.059704+0000","last_change":"2023-07-14T17:42:42.543586+0000","last_active":"2023-07-14T17:42:48.059704+0000","last_peered":"2023-07-14T17:42:48.059704+0000","last_clean":"2023-07-14T17:42:48.059704+0000","last_became_active":"2023-07-14T17:42:42.543259+0000","last_became_peered":"2023-07-14T17:42:42.543259+0000","last_unstale":"2023-07-14T17:42:48.059704+0000","last_undegraded":"2023-07-14T17:42:48.059704+0000","last_fullsized":"2023-07-14T17:42:48.059704+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:24:35.934043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65561,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":438,"num_read_kb":292,"num_write":234,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":196683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":206,"ondisk_log_size":206,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":467172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.495},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.44}]},{"osd":1,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.61399999999999999}]},{"osd":2,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.46800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":4,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.65900000000000003}]},{"osd":5,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.63},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.64300000000000002}]},{"osd":6,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.748},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.72099999999999997}]},{"osd":7,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.70999999999999996}]},{"osd":8,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.76600000000000001}]},{"osd":9,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.82399999999999995}]},{"osd":10,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.81000000000000005}]}]},{"osd":10,"up_from":69,"seq":296352743452,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28652,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716500,"statfs":{"total":95995035648,"available":95965696000,"internally_reserved":0,"allocated":1671168,"data_stored":1122439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":1,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.47699999999999998}]},{"osd":2,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.92100000000000004}]},{"osd":3,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.72099999999999997}]},{"osd":4,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.51300000000000001}]},{"osd":5,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.82899999999999996}]},{"osd":6,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.86699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":8,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.73599999999999999}]},{"osd":9,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.76600000000000001}]},{"osd":11,"last update":"Fri Jul 14 17:43:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.93200000000000005}]}]},{"osd":9,"up_from":64,"seq":274877906976,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":471246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.41099999999999998,"15min":0.42399999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.497,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.45400000000000001,"15min":0.47199999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.55800000000000005,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.39400000000000002}]},{"osd":1,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.59099999999999997,"15min":0.60599999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.69899999999999995,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.62},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.60199999999999998,"15min":0.61599999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70999999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.499}]},{"osd":2,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79900000000000004,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.64500000000000002,"15min":0.65700000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.73199999999999998,"5min":1.119,"15min":1.119},"last":0.64500000000000002}]},{"osd":3,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.48999999999999999,"15min":0.50600000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.52700000000000002,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.49199999999999999,"15min":0.505},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64400000000000002,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.34599999999999997}]},{"osd":4,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.628,"15min":0.64300000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.70699999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.63600000000000001,"15min":0.64800000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.68300000000000005,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.63400000000000001}]},{"osd":5,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.64600000000000002,"15min":0.65800000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.78400000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.65800000000000003,"15min":0.67100000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79600000000000004,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.66700000000000004}]},{"osd":6,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.372},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.52500000000000002}]},{"osd":7,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.70599999999999996,"15min":0.72799999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71099999999999997,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.71699999999999997,"15min":0.73799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.79400000000000004,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.57399999999999995}]},{"osd":8,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.749,"15min":0.76800000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.82699999999999996,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.74199999999999999,"15min":0.75800000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.85899999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.748}]},{"osd":10,"last update":"Fri Jul 14 17:43:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.73399999999999999,"15min":0.755},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.74399999999999999,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.73199999999999998,"15min":0.748},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.73999999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.65800000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]}]},{"osd":8,"up_from":57,"seq":244813135909,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":467528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.441},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53400000000000003,"15min":0.53300000000000003},"min":{"1min":0.35499999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.64100000000000001}]},{"osd":1,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.372,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83899999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.40999999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.39700000000000002}]},{"osd":2,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45800000000000002,"15min":0.46100000000000002},"min":{"1min":0.27500000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60599999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.47199999999999998,"15min":0.47999999999999998},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56499999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.54700000000000004}]},{"osd":3,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.36299999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.46000000000000002}]},{"osd":4,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85099999999999998,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64300000000000002,"15min":0.64200000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89700000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.53100000000000003}]},{"osd":5,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57399999999999995,"15min":0.58099999999999996},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72499999999999998,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.312,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.75,"5min":1.018,"15min":1.018},"last":0.33700000000000002}]},{"osd":6,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.66600000000000004,"15min":0.66800000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68500000000000005,"15min":0.69299999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84799999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.66400000000000003}]},{"osd":7,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.51000000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92400000000000004,"5min":1.008,"15min":1.008},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71099999999999997,"15min":0.70999999999999996},"min":{"1min":0.48899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88200000000000001,"5min":1.258,"15min":1.258},"last":0.74099999999999999}]},{"osd":9,"last update":"Fri Jul 14 17:43:19 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.68500000000000005,"15min":0.67000000000000004},"min":{"1min":0.441,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.68600000000000005,"15min":0.67200000000000004},"min":{"1min":0.46899999999999997,"5min":0.44,"15min":0.44},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.76000000000000001}]},{"osd":10,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.69699999999999995,"15min":0.68100000000000005},"min":{"1min":0.57199999999999995,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94399999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.70699999999999996,"15min":0.68500000000000005},"min":{"1min":0.61099999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.79700000000000004}]},{"osd":11,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.82499999999999996}]}]},{"osd":7,"up_from":49,"seq":210453397544,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28560,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716592,"statfs":{"total":95995035648,"available":95965790208,"internally_reserved":0,"allocated":1581056,"data_stored":1064270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.503},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.59499999999999997}]},{"osd":1,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.41499999999999998,"15min":0.42299999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.57799999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.46000000000000002,"15min":0.47099999999999997},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.51800000000000002,"5min":0.746,"15min":0.746},"last":0.57999999999999996}]},{"osd":2,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.628,"15min":0.60299999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":5.1050000000000004,"5min":5.1050000000000004,"15min":5.1050000000000004},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.71999999999999997,"5min":1.048,"15min":1.048},"last":0.66700000000000004}]},{"osd":3,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.58699999999999997,"15min":0.59299999999999997},"min":{"1min":0.436,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.755},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.59899999999999998,"15min":0.61199999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72499999999999998,"5min":0.995,"15min":0.995},"last":0.80700000000000005}]},{"osd":4,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.49299999999999999,"15min":0.499},"min":{"1min":0.312,"5min":0.31,"15min":0.31},"max":{"1min":0.59999999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.52000000000000002,"15min":0.53200000000000003},"min":{"1min":0.28899999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.66900000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.72999999999999998}]},{"osd":5,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.63400000000000001,"15min":0.63200000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64100000000000001,"15min":0.64300000000000002},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.94199999999999995,"5min":1.02,"15min":1.02},"last":0.89100000000000001}]},{"osd":6,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.64000000000000001,"15min":0.65000000000000002},"min":{"1min":0.435,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.878},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.84999999999999998}]},{"osd":8,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.45700000000000002,"5min":0.441,"15min":0.441},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.71299999999999997,"15min":0.72099999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0620000000000001,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.90600000000000003}]},{"osd":9,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.69399999999999995,"15min":0.70199999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.01,"5min":1.196,"15min":1.196},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.91200000000000003,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.86599999999999999}]},{"osd":10,"last update":"Fri Jul 14 17:42:42 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.83199999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]}]},{"osd":6,"up_from":44,"seq":188978561069,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":472116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.36799999999999999,"15min":0.35499999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.28199999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.307}]},{"osd":1,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57999999999999996,"15min":0.56799999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.76000000000000001,"5min":0.87,"15min":0.87},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.65000000000000002,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46000000000000002}]},{"osd":2,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59399999999999997,"15min":0.58999999999999997},"min":{"1min":0.44500000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.56399999999999995}]},{"osd":3,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.44400000000000001,"15min":0.44700000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68500000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.38},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.47199999999999998,"15min":0.45400000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78900000000000003,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.40400000000000003}]},{"osd":4,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85299999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.60799999999999998,"15min":0.60599999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81799999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.435}]},{"osd":5,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.60099999999999998,"15min":0.59299999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.68600000000000005,"15min":0.66300000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.8679999999999999,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.624}]},{"osd":7,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.754,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.66600000000000004,"15min":0.68000000000000005},"min":{"1min":0.47999999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.73499999999999999,"5min":1.012,"15min":1.012},"last":0.55500000000000005}]},{"osd":8,"last update":"Fri Jul 14 17:43:04 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.73599999999999999,"15min":0.755},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.98999999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.69499999999999995,"15min":0.71099999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92500000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.57399999999999995}]},{"osd":9,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.68100000000000005,"15min":0.70099999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.91500000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.65700000000000003,"15min":0.67700000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.90200000000000002,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.52300000000000002}]},{"osd":10,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.72799999999999998,"15min":0.746},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.81999999999999995,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.755,"15min":0.76900000000000002},"min":{"1min":0.52500000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.83799999999999997,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.64000000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":531697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.503,"15min":0.46700000000000003},"min":{"1min":0.33800000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.51500000000000001,"15min":0.47199999999999998},"min":{"1min":0.33600000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.78600000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Jul 14 17:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.54200000000000004,"15min":0.497},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.86099999999999999,"5min":0.879,"15min":0.879},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.52300000000000002,"15min":0.48899999999999999},"min":{"1min":0.36499999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.79200000000000004}]},{"osd":3,"last update":"Fri Jul 14 17:43:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53400000000000003,"15min":0.51600000000000001},"min":{"1min":0.31,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.75600000000000001,"5min":0.873,"15min":0.873},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57699999999999996,"15min":0.54500000000000004},"min":{"1min":0.44700000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.752,"5min":0.873,"15min":0.873},"last":0.63700000000000001}]},{"osd":4,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.45400000000000001,"15min":0.42999999999999999},"min":{"1min":0.34699999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.45700000000000002,"15min":0.433},"min":{"1min":0.29399999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.51900000000000002}]},{"osd":5,"last update":"Fri Jul 14 17:42:59 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.60899999999999999,"15min":0.58499999999999996},"min":{"1min":0.48299999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.58199999999999996,"15min":0.57099999999999995},"min":{"1min":0.42699999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.73399999999999999}]},{"osd":6,"last update":"Fri Jul 14 17:43:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.41599999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.66200000000000003}]},{"osd":7,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.56899999999999995,"15min":0.52800000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.55300000000000005,"15min":0.53300000000000003},"min":{"1min":0.373,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.70099999999999996}]},{"osd":8,"last update":"Fri Jul 14 17:42:59 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.65900000000000003,"15min":0.65600000000000003},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1,"5min":1,"15min":1},"last":0.75},{"interface":"front","average":{"1min":0.746,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.54000000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.88100000000000001}]},{"osd":9,"last update":"Fri Jul 14 17:43:19 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65500000000000003,"15min":0.65700000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.88400000000000001,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70199999999999996,"15min":0.70099999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.80800000000000005}]},{"osd":10,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.84299999999999997}]},{"osd":11,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.89500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":464062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 17:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.61499999999999999,"15min":0.53700000000000003},"min":{"1min":0.38500000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":8.5199999999999996,"5min":8.5199999999999996,"15min":8.5199999999999996},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.0960000000000001,"5min":0.65800000000000003,"15min":0.56000000000000005},"min":{"1min":0.41599999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":10.31,"5min":10.31,"15min":10.31},"last":0.46600000000000003}]},{"osd":2,"last update":"Fri Jul 14 17:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55900000000000005,"15min":0.53400000000000003},"min":{"1min":0.40999999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.502},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56999999999999995,"15min":0.55300000000000005},"min":{"1min":0.379,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.39200000000000002}]},{"osd":3,"last update":"Fri Jul 14 17:41:15 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44700000000000001,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.40999999999999998,"15min":0.40600000000000003},"min":{"1min":0.27900000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.61699999999999999,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.40799999999999997}]},{"osd":4,"last update":"Fri Jul 14 17:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55700000000000005,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.121,"5min":0.68100000000000005,"15min":0.58099999999999996},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":10.374000000000001,"5min":10.374000000000001,"15min":10.374000000000001},"last":0.54600000000000004}]},{"osd":5,"last update":"Fri Jul 14 17:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56499999999999995,"15min":0.55100000000000005},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.755,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59599999999999997,"15min":0.58999999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84099999999999997,"5min":0.88,"15min":0.88},"last":0.48899999999999999}]},{"osd":6,"last update":"Fri Jul 14 17:43:18 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.57999999999999996,"15min":0.502},"min":{"1min":0.30599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":10.468999999999999,"5min":10.468999999999999,"15min":10.468999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.58499999999999996,"15min":0.51600000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":8.4350000000000005,"5min":8.4350000000000005,"15min":8.4350000000000005},"last":0.377}]},{"osd":7,"last update":"Fri Jul 14 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.56399999999999995}]},{"osd":8,"last update":"Fri Jul 14 17:43:06 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.68500000000000005,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.82799999999999996,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.67100000000000004,"15min":0.67900000000000005},"min":{"1min":0.46200000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.89900000000000002,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.45200000000000001}]},{"osd":9,"last update":"Fri Jul 14 17:43:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.77600000000000002,"15min":0.72799999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":10.343,"5min":10.343,"15min":10.343},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.5509999999999999,"5min":0.89000000000000001,"15min":0.78000000000000003},"min":{"1min":0.45300000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":18.739000000000001,"5min":18.739000000000001,"15min":18.739000000000001},"last":0.439}]},{"osd":10,"last update":"Fri Jul 14 17:42:44 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.57399999999999995}]},{"osd":11,"last update":"Fri Jul 14 17:43:06 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.53400000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":541552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.47499999999999998},"min":{"1min":0.41099999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.48299999999999998,"15min":0.46700000000000003},"min":{"1min":0.39100000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.59799999999999998}]},{"osd":1,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.53100000000000003,"15min":0.48299999999999998},"min":{"1min":0.35199999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.55300000000000005,"15min":0.50900000000000001},"min":{"1min":0.5,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.80100000000000005}]},{"osd":3,"last update":"Fri Jul 14 17:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.57499999999999996,"15min":0.54700000000000004},"min":{"1min":0.38100000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55900000000000005,"15min":0.53900000000000003},"min":{"1min":0.40100000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.86899999999999999,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.76900000000000002}]},{"osd":4,"last update":"Fri Jul 14 17:41:35 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.51900000000000002,"15min":0.51300000000000001},"min":{"1min":0.38400000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.55400000000000005,"15min":0.54600000000000004},"min":{"1min":0.39900000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.73799999999999999}]},{"osd":5,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.45600000000000002,"15min":0.43099999999999999},"min":{"1min":0.30399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.878},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.45200000000000001,"15min":0.437},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.89800000000000002}]},{"osd":6,"last update":"Fri Jul 14 17:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.60099999999999998,"15min":0.58099999999999996},"min":{"1min":0.47199999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.76000000000000001,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59899999999999998,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81799999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.85499999999999998}]},{"osd":7,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.59099999999999997,"15min":0.57499999999999996},"min":{"1min":0.40200000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.66000000000000003,"15min":0.64600000000000002},"min":{"1min":0.52500000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":1.0620000000000001}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.61299999999999999,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":1.006},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.59299999999999997,"15min":0.57399999999999995},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.96299999999999997}]},{"osd":9,"last update":"Fri Jul 14 17:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71399999999999997,"15min":0.70999999999999996},"min":{"1min":0.56999999999999995,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.995},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.76700000000000002,"15min":0.75600000000000001},"min":{"1min":0.64100000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":1.034}]},{"osd":10,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":1.046}]},{"osd":11,"last update":"Fri Jul 14 17:43:40 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":1.089},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":1.0209999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":462362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:16 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34300000000000003,"15min":0.32300000000000001},"min":{"1min":0.26900000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.437,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.36099999999999999,"15min":0.33200000000000002},"min":{"1min":0.28100000000000003,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.317}]},{"osd":1,"last update":"Fri Jul 14 17:43:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54200000000000004,"15min":0.50600000000000001},"min":{"1min":0.38600000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.82599999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54200000000000004,"15min":0.505},"min":{"1min":0.33300000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72499999999999998,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.44600000000000001}]},{"osd":2,"last update":"Fri Jul 14 17:43:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54700000000000004,"15min":0.52400000000000002},"min":{"1min":0.41899999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.69699999999999995,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.52800000000000002},"min":{"1min":0.40899999999999997,"5min":0.379,"15min":0.379},"max":{"1min":0.71999999999999997,"5min":1.423,"15min":1.423},"last":0.53300000000000003}]},{"osd":4,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58099999999999996,"15min":0.56699999999999995},"min":{"1min":0.37,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.71299999999999997,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.378},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.60299999999999998,"15min":0.58199999999999996},"min":{"1min":0.50600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84199999999999997,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.42499999999999999}]},{"osd":5,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.38100000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85299999999999998,"5min":1.593,"15min":1.593},"last":0.504},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.45400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.85699999999999998,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.46400000000000002}]},{"osd":6,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.52600000000000002,"15min":0.52900000000000003},"min":{"1min":0.32800000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.73699999999999999,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.46899999999999997,"15min":0.47599999999999998},"min":{"1min":0.32700000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.57399999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.33200000000000002}]},{"osd":7,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.63700000000000001,"15min":0.64300000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79300000000000004,"5min":1.28,"15min":1.28},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.65200000000000002,"15min":0.64400000000000002},"min":{"1min":0.56200000000000006,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.89000000000000001,"5min":1.03,"15min":1.03},"last":0.54400000000000004}]},{"osd":8,"last update":"Fri Jul 14 17:41:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.55800000000000005}]},{"osd":9,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.878,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.68100000000000005,"15min":0.69699999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.55700000000000005}]},{"osd":10,"last update":"Fri Jul 14 17:42:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.56799999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]}]},{"osd":4,"up_from":32,"seq":137438953525,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":467070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.35399999999999998,"5min":0.311,"15min":0.311},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.52100000000000002,"15min":0.502},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.496}]},{"osd":1,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.41699999999999998,"15min":0.38200000000000001},"min":{"1min":0.372,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.61699999999999999,"5min":0.755,"15min":0.755},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.42699999999999999,"15min":0.40000000000000002},"min":{"1min":0.38,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.621,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.47799999999999998}]},{"osd":2,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.57499999999999996,"15min":0.56299999999999994},"min":{"1min":0.496,"5min":0.38,"15min":0.38},"max":{"1min":0.80200000000000005,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.622,"15min":0.58199999999999996},"min":{"1min":0.54400000000000004,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89600000000000002,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.86099999999999999}]},{"osd":3,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.59799999999999998,"15min":0.55100000000000005},"min":{"1min":0.44600000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.86599999999999999,"5min":1.498,"15min":1.498},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56200000000000006,"15min":0.53400000000000003},"min":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81100000000000005,"5min":1.629,"15min":1.629},"last":0.55000000000000004}]},{"osd":5,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.63100000000000001,"15min":0.61099999999999999},"min":{"1min":0.51500000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89800000000000002,"5min":1.256,"15min":1.256},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.624,"15min":0.60999999999999999},"min":{"1min":0.44900000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.68000000000000005}]},{"osd":6,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.72199999999999998,"15min":0.746},"min":{"1min":0.47699999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91800000000000004,"5min":2.7789999999999999,"15min":2.7789999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68200000000000005,"15min":0.66700000000000004},"min":{"1min":0.46600000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.95899999999999996,"5min":1.476,"15min":1.476},"last":0.71399999999999997}]},{"osd":7,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.57899999999999996,"15min":0.53300000000000003},"min":{"1min":0.49099999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.97299999999999998,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.61399999999999999,"15min":0.58399999999999996},"min":{"1min":0.5,"5min":0.377,"15min":0.377},"max":{"1min":0.88500000000000001,"5min":1.002,"15min":1.002},"last":0.73799999999999999}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.65600000000000003,"15min":0.64400000000000002},"min":{"1min":0.41999999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72299999999999998,"15min":0.71399999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.82699999999999996}]},{"osd":9,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.69399999999999995,"15min":0.69099999999999995},"min":{"1min":0.54000000000000004,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88100000000000001,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.80100000000000005,"15min":0.80400000000000005},"min":{"1min":0.61899999999999999,"5min":0.502,"15min":0.502},"max":{"1min":1.0049999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.78400000000000003}]},{"osd":10,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.82899999999999996,"15min":0.83099999999999996},"min":{"1min":0.63400000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0600000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.748,"5min":0.73199999999999998,"15min":0.72999999999999998},"min":{"1min":0.59799999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.75800000000000001}]},{"osd":11,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.93999999999999995}]}]},{"osd":5,"up_from":37,"seq":158913790002,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28588,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716564,"statfs":{"total":95995035648,"available":95965761536,"internally_reserved":0,"allocated":1605632,"data_stored":1069068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47099999999999997,"15min":0.44700000000000001},"min":{"1min":0.35499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77800000000000002,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.495},{"interface":"front","average":{"1min":0.501,"5min":0.48299999999999998,"15min":0.45700000000000002},"min":{"1min":0.35599999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65400000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.47499999999999998}]},{"osd":1,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.55100000000000005,"15min":0.497},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":5.3630000000000004,"5min":5.3630000000000004,"15min":5.3630000000000004},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51300000000000001,"15min":0.48999999999999999},"min":{"1min":0.42399999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.58599999999999997}]},{"osd":2,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.33500000000000002},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.379},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.376,"15min":0.35999999999999999},"min":{"1min":0.30499999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.40899999999999997}]},{"osd":3,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.501,"15min":0.495},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.51800000000000002,"15min":0.495},"min":{"1min":0.38700000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.747,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.60999999999999999}]},{"osd":4,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.53500000000000003,"15min":0.50800000000000001},"min":{"1min":0.434,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78100000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54300000000000004,"15min":0.52000000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.75600000000000001}]},{"osd":6,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.39000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.75700000000000001,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.627},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.56699999999999995,"15min":0.54300000000000004},"min":{"1min":0.46400000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80800000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.67900000000000005}]},{"osd":7,"last update":"Fri Jul 14 17:43:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.58899999999999997,"15min":0.57399999999999995},"min":{"1min":0.47599999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.63300000000000001,"15min":0.61299999999999999},"min":{"1min":0.51200000000000001,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.73899999999999999}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.54700000000000004,"15min":0.54000000000000004},"min":{"1min":0.38100000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.754,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.372,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.876,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.81299999999999994}]},{"osd":9,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.56999999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91000000000000003,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68400000000000005,"15min":0.67600000000000005},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.69399999999999995}]},{"osd":10,"last update":"Fri Jul 14 17:42:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":5.3330000000000002,"5min":5.3330000000000002,"15min":5.3330000000000002},"last":0.77700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":60629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T17:43:46.408 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T17:43:46.969 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:47.163 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:46 smithi178 ceph-mon[129368]: pgmap v289: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2023-07-14T17:43:47.164 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:46 smithi178 ceph-mon[129368]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:43:47.184 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:46 smithi088 ceph-mon[127465]: pgmap v289: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2023-07-14T17:43:47.184 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:46 smithi088 ceph-mon[127465]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:43:47.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:46 smithi107 ceph-mon[128999]: pgmap v289: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2023-07-14T17:43:47.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:46 smithi107 ceph-mon[128999]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:43:48.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.186+0000 7f6b1fc85700 1 -- 172.21.15.88:0/3548553376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b18103ee0 msgr2=0x7f6b1810ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:48.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.186+0000 7f6b1fc85700 1 --2- 172.21.15.88:0/3548553376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b18103ee0 0x7f6b1810ba60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000b0d0 tx=0x7f6b1000b3e0 comp rx=0 tx=0).stop 2023-07-14T17:43:48.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.186+0000 7f6b1fc85700 1 -- 172.21.15.88:0/3548553376 shutdown_connections 2023-07-14T17:43:48.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.186+0000 7f6b1fc85700 1 --2- 172.21.15.88:0/3548553376 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b18103ee0 0x7f6b1810ba60 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:48.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.186+0000 7f6b1fc85700 1 --2- 172.21.15.88:0/3548553376 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b180fee40 0x7f6b180ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:48.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.186+0000 7f6b1fc85700 1 --2- 172.21.15.88:0/3548553376 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b180ff7e0 0x7f6b181037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:48.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.186+0000 7f6b1fc85700 1 -- 172.21.15.88:0/3548553376 >> 172.21.15.88:0/3548553376 conn(0x7f6b18075ca0 msgr2=0x7f6b180760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:48.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.186+0000 7f6b1fc85700 1 -- 172.21.15.88:0/3548553376 shutdown_connections 2023-07-14T17:43:48.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.186+0000 7f6b1fc85700 1 -- 172.21.15.88:0/3548553376 wait complete. 2023-07-14T17:43:48.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.187+0000 7f6b1fc85700 1 Processor -- start 2023-07-14T17:43:48.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.187+0000 7f6b1fc85700 1 -- start start 2023-07-14T17:43:48.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.187+0000 7f6b1fc85700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b180fee40 0x7f6b18196bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:48.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.187+0000 7f6b1fc85700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b180ff7e0 0x7f6b181970f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:48.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.187+0000 7f6b1fc85700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b18103ee0 0x7f6b181a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:48.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.187+0000 7f6b1fc85700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b181a2170 con 0x7f6b18103ee0 2023-07-14T17:43:48.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.187+0000 7f6b1fc85700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b181a22b0 con 0x7f6b180ff7e0 2023-07-14T17:43:48.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.187+0000 7f6b1fc85700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b181a23f0 con 0x7f6b180fee40 2023-07-14T17:43:48.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.187+0000 7f6b1e222700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b18103ee0 0x7f6b181a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:48.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.187+0000 7f6b1e222700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b18103ee0 0x7f6b181a1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42454/0 (socket says 172.21.15.88:42454) 2023-07-14T17:43:48.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.187+0000 7f6b1e222700 1 -- 172.21.15.88:0/3042706868 learned_addr learned my addr 172.21.15.88:0/3042706868 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:48.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.187+0000 7f6b1da21700 1 --2- 172.21.15.88:0/3042706868 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b180fee40 0x7f6b18196bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:48.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.187+0000 7f6b1d220700 1 --2- 172.21.15.88:0/3042706868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b180ff7e0 0x7f6b181970f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:48.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.188+0000 7f6b1e222700 1 -- 172.21.15.88:0/3042706868 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b180fee40 msgr2=0x7f6b18196bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:48.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.188+0000 7f6b1e222700 1 --2- 172.21.15.88:0/3042706868 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b180fee40 0x7f6b18196bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:48.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.188+0000 7f6b1e222700 1 -- 172.21.15.88:0/3042706868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b180ff7e0 msgr2=0x7f6b181970f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:48.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.188+0000 7f6b1e222700 1 --2- 172.21.15.88:0/3042706868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b180ff7e0 0x7f6b181970f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:48.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.188+0000 7f6b1e222700 1 -- 172.21.15.88:0/3042706868 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b10009d00 con 0x7f6b18103ee0 2023-07-14T17:43:48.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.188+0000 7f6b1e222700 1 --2- 172.21.15.88:0/3042706868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b18103ee0 0x7f6b181a1c30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6b10009fd0 tx=0x7f6b10009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:48.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.188+0000 7f6b0effd700 1 -- 172.21.15.88:0/3042706868 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1000e040 con 0x7f6b18103ee0 2023-07-14T17:43:48.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.188+0000 7f6b0effd700 1 -- 172.21.15.88:0/3042706868 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6b10004e40 con 0x7f6b18103ee0 2023-07-14T17:43:48.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.188+0000 7f6b1fc85700 1 -- 172.21.15.88:0/3042706868 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b181a25f0 con 0x7f6b18103ee0 2023-07-14T17:43:48.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.188+0000 7f6b0effd700 1 -- 172.21.15.88:0/3042706868 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1001b970 con 0x7f6b18103ee0 2023-07-14T17:43:48.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.188+0000 7f6b1fc85700 1 -- 172.21.15.88:0/3042706868 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b181a2a10 con 0x7f6b18103ee0 2023-07-14T17:43:48.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.189+0000 7f6b1fc85700 1 -- 172.21.15.88:0/3042706868 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b1804e340 con 0x7f6b18103ee0 2023-07-14T17:43:48.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.191+0000 7f6b0effd700 1 -- 172.21.15.88:0/3042706868 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6b1001bad0 con 0x7f6b18103ee0 2023-07-14T17:43:48.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.191+0000 7f6b0effd700 1 --2- 172.21.15.88:0/3042706868 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6b040a11e0 0x7f6b040a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:48.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.191+0000 7f6b1da21700 1 --2- 172.21.15.88:0/3042706868 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6b040a11e0 0x7f6b040a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:48.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.191+0000 7f6b0effd700 1 -- 172.21.15.88:0/3042706868 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6b100cebc0 con 0x7f6b18103ee0 2023-07-14T17:43:48.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.192+0000 7f6b1da21700 1 --2- 172.21.15.88:0/3042706868 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6b040a11e0 0x7f6b040a3690 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6b14005950 tx=0x7f6b1400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:48.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.195+0000 7f6b0effd700 1 -- 172.21.15.88:0/3042706868 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6b10097f30 con 0x7f6b18103ee0 2023-07-14T17:43:48.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.383+0000 7f6b1fc85700 1 -- 172.21.15.88:0/3042706868 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6b180fcc00 con 0x7f6b040a11e0 2023-07-14T17:43:48.376 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.395+0000 7f6b0effd700 1 -- 172.21.15.88:0/3042706868 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+425642 (secure 0 0 0) 0x7f6b180fcc00 con 0x7f6b040a11e0 2023-07-14T17:43:48.380 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:43:48.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.398+0000 7f6b1fc85700 1 -- 172.21.15.88:0/3042706868 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6b040a11e0 msgr2=0x7f6b040a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:48.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.398+0000 7f6b1fc85700 1 --2- 172.21.15.88:0/3042706868 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6b040a11e0 0x7f6b040a3690 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6b14005950 tx=0x7f6b1400a300 comp rx=0 tx=0).stop 2023-07-14T17:43:48.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.398+0000 7f6b1fc85700 1 -- 172.21.15.88:0/3042706868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b18103ee0 msgr2=0x7f6b181a1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:48.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.398+0000 7f6b1fc85700 1 --2- 172.21.15.88:0/3042706868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b18103ee0 0x7f6b181a1c30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6b10009fd0 tx=0x7f6b10009380 comp rx=0 tx=0).stop 2023-07-14T17:43:48.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.398+0000 7f6b1fc85700 1 -- 172.21.15.88:0/3042706868 shutdown_connections 2023-07-14T17:43:48.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.398+0000 7f6b1fc85700 1 --2- 172.21.15.88:0/3042706868 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6b040a11e0 0x7f6b040a3690 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:48.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.398+0000 7f6b1fc85700 1 --2- 172.21.15.88:0/3042706868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6b18103ee0 0x7f6b181a1c30 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:48.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.398+0000 7f6b1fc85700 1 --2- 172.21.15.88:0/3042706868 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6b180fee40 0x7f6b18196bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:48.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.398+0000 7f6b1fc85700 1 --2- 172.21.15.88:0/3042706868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6b180ff7e0 0x7f6b181970f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:48.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.398+0000 7f6b1fc85700 1 -- 172.21.15.88:0/3042706868 >> 172.21.15.88:0/3042706868 conn(0x7f6b18075ca0 msgr2=0x7f6b180fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:48.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.398+0000 7f6b1fc85700 1 -- 172.21.15.88:0/3042706868 shutdown_connections 2023-07-14T17:43:48.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:48.398+0000 7f6b1fc85700 1 -- 172.21.15.88:0/3042706868 wait complete. 2023-07-14T17:43:48.386 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-07-14T17:43:48.407 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:48 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:48.408 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:48 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:48.408 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:48 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:48.408 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:48 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:48.409 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:48 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:48.409 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:48 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:48.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:48 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:48.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:48 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:48.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:48 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:48.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:48 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:48.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:48 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:48.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:48 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:48.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:48 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:48.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:48 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:48.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:48 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:48.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:48 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:48.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:48 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:48.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:48 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:48.826 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2023-07-14T17:43:47.887768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656175,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":569,"num_read_kb":401,"num_write":417,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":345,"ondisk_log_size":345,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":422,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337832,"kb_used_data":13600,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124603992,"statfs":{"total":1151940427776,"available":1151594487808,"internally_reserved":0,"allocated":13926400,"data_stored":7657810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":90,"num_read_kb":45,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002293"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892404+0000","last_change":"2023-07-14T17:42:42.540396+0000","last_active":"2023-07-14T17:43:44.892404+0000","last_peered":"2023-07-14T17:43:44.892404+0000","last_clean":"2023-07-14T17:43:44.892404+0000","last_became_active":"2023-07-14T17:42:42.539732+0000","last_became_peered":"2023-07-14T17:42:42.539732+0000","last_unstale":"2023-07-14T17:43:44.892404+0000","last_undegraded":"2023-07-14T17:43:44.892404+0000","last_fullsized":"2023-07-14T17:43:44.892404+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:45:35.673679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313254+0000","last_change":"2023-07-14T17:42:42.532153+0000","last_active":"2023-07-14T17:42:48.313254+0000","last_peered":"2023-07-14T17:42:48.313254+0000","last_clean":"2023-07-14T17:42:48.313254+0000","last_became_active":"2023-07-14T17:42:42.531745+0000","last_became_peered":"2023-07-14T17:42:42.531745+0000","last_unstale":"2023-07-14T17:42:48.313254+0000","last_undegraded":"2023-07-14T17:42:48.313254+0000","last_fullsized":"2023-07-14T17:42:48.313254+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:55.889251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313184+0000","last_change":"2023-07-14T17:42:42.549497+0000","last_active":"2023-07-14T17:42:48.313184+0000","last_peered":"2023-07-14T17:42:48.313184+0000","last_clean":"2023-07-14T17:42:48.313184+0000","last_became_active":"2023-07-14T17:42:42.549201+0000","last_became_peered":"2023-07-14T17:42:42.549201+0000","last_unstale":"2023-07-14T17:42:48.313184+0000","last_undegraded":"2023-07-14T17:42:48.313184+0000","last_fullsized":"2023-07-14T17:42:48.313184+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:20:16.806348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542180+0000","last_change":"2023-07-14T17:42:42.543460+0000","last_active":"2023-07-14T17:42:47.542180+0000","last_peered":"2023-07-14T17:42:47.542180+0000","last_clean":"2023-07-14T17:42:47.542180+0000","last_became_active":"2023-07-14T17:42:42.543127+0000","last_became_peered":"2023-07-14T17:42:42.543127+0000","last_unstale":"2023-07-14T17:42:47.542180+0000","last_undegraded":"2023-07-14T17:42:47.542180+0000","last_fullsized":"2023-07-14T17:42:47.542180+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:40:04.077779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835392+0000","last_change":"2023-07-14T17:42:42.534242+0000","last_active":"2023-07-14T17:42:47.835392+0000","last_peered":"2023-07-14T17:42:47.835392+0000","last_clean":"2023-07-14T17:42:47.835392+0000","last_became_active":"2023-07-14T17:42:42.533536+0000","last_became_peered":"2023-07-14T17:42:42.533536+0000","last_unstale":"2023-07-14T17:42:47.835392+0000","last_undegraded":"2023-07-14T17:42:47.835392+0000","last_fullsized":"2023-07-14T17:42:47.835392+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:57:39.927450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896541+0000","last_change":"2023-07-14T17:42:42.538042+0000","last_active":"2023-07-14T17:43:44.896541+0000","last_peered":"2023-07-14T17:43:44.896541+0000","last_clean":"2023-07-14T17:43:44.896541+0000","last_became_active":"2023-07-14T17:42:42.537612+0000","last_became_peered":"2023-07-14T17:42:42.537612+0000","last_unstale":"2023-07-14T17:43:44.896541+0000","last_undegraded":"2023-07-14T17:43:44.896541+0000","last_fullsized":"2023-07-14T17:43:44.896541+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:29:37.412187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894322+0000","last_change":"2023-07-14T17:42:42.545318+0000","last_active":"2023-07-14T17:43:44.894322+0000","last_peered":"2023-07-14T17:43:44.894322+0000","last_clean":"2023-07-14T17:43:44.894322+0000","last_became_active":"2023-07-14T17:42:42.544984+0000","last_became_peered":"2023-07-14T17:42:42.544984+0000","last_unstale":"2023-07-14T17:43:44.894322+0000","last_undegraded":"2023-07-14T17:43:44.894322+0000","last_fullsized":"2023-07-14T17:43:44.894322+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:10:59.223910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.539255+0000","last_change":"2023-07-14T17:42:42.532609+0000","last_active":"2023-07-14T17:42:47.539255+0000","last_peered":"2023-07-14T17:42:47.539255+0000","last_clean":"2023-07-14T17:42:47.539255+0000","last_became_active":"2023-07-14T17:42:42.532270+0000","last_became_peered":"2023-07-14T17:42:42.532270+0000","last_unstale":"2023-07-14T17:42:47.539255+0000","last_undegraded":"2023-07-14T17:42:47.539255+0000","last_fullsized":"2023-07-14T17:42:47.539255+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:20:00.572052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T17:43:43.897259+0000","last_change":"2023-07-14T17:42:42.548973+0000","last_active":"2023-07-14T17:43:43.897259+0000","last_peered":"2023-07-14T17:43:43.897259+0000","last_clean":"2023-07-14T17:43:43.897259+0000","last_became_active":"2023-07-14T17:42:42.548672+0000","last_became_peered":"2023-07-14T17:42:42.548672+0000","last_unstale":"2023-07-14T17:43:43.897259+0000","last_undegraded":"2023-07-14T17:43:43.897259+0000","last_fullsized":"2023-07-14T17:43:43.897259+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:45:00.580622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896875+0000","last_change":"2023-07-14T17:42:42.540969+0000","last_active":"2023-07-14T17:43:44.896875+0000","last_peered":"2023-07-14T17:43:44.896875+0000","last_clean":"2023-07-14T17:43:44.896875+0000","last_became_active":"2023-07-14T17:42:42.540653+0000","last_became_peered":"2023-07-14T17:42:42.540653+0000","last_unstale":"2023-07-14T17:43:44.896875+0000","last_undegraded":"2023-07-14T17:43:44.896875+0000","last_fullsized":"2023-07-14T17:43:44.896875+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:07.062764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894491+0000","last_change":"2023-07-14T17:42:42.550016+0000","last_active":"2023-07-14T17:43:44.894491+0000","last_peered":"2023-07-14T17:43:44.894491+0000","last_clean":"2023-07-14T17:43:44.894491+0000","last_became_active":"2023-07-14T17:42:42.549653+0000","last_became_peered":"2023-07-14T17:42:42.549653+0000","last_unstale":"2023-07-14T17:43:44.894491+0000","last_undegraded":"2023-07-14T17:43:44.894491+0000","last_fullsized":"2023-07-14T17:43:44.894491+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:43.271416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542109+0000","last_change":"2023-07-14T17:42:42.552105+0000","last_active":"2023-07-14T17:42:47.542109+0000","last_peered":"2023-07-14T17:42:47.542109+0000","last_clean":"2023-07-14T17:42:47.542109+0000","last_became_active":"2023-07-14T17:42:42.550830+0000","last_became_peered":"2023-07-14T17:42:42.550830+0000","last_unstale":"2023-07-14T17:42:47.542109+0000","last_undegraded":"2023-07-14T17:42:47.542109+0000","last_fullsized":"2023-07-14T17:42:47.542109+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:47:37.896520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324361+0000","last_change":"2023-07-14T17:42:42.534963+0000","last_active":"2023-07-14T17:43:45.324361+0000","last_peered":"2023-07-14T17:43:45.324361+0000","last_clean":"2023-07-14T17:43:45.324361+0000","last_became_active":"2023-07-14T17:42:42.534546+0000","last_became_peered":"2023-07-14T17:42:42.534546+0000","last_unstale":"2023-07-14T17:43:45.324361+0000","last_undegraded":"2023-07-14T17:43:45.324361+0000","last_fullsized":"2023-07-14T17:43:45.324361+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:37:07.860214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.312996+0000","last_change":"2023-07-14T17:42:42.530073+0000","last_active":"2023-07-14T17:42:48.312996+0000","last_peered":"2023-07-14T17:42:48.312996+0000","last_clean":"2023-07-14T17:42:48.312996+0000","last_became_active":"2023-07-14T17:42:42.529767+0000","last_became_peered":"2023-07-14T17:42:42.529767+0000","last_unstale":"2023-07-14T17:42:48.312996+0000","last_undegraded":"2023-07-14T17:42:48.312996+0000","last_fullsized":"2023-07-14T17:42:48.312996+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:13:05.025890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.312868+0000","last_change":"2023-07-14T17:42:42.548168+0000","last_active":"2023-07-14T17:42:48.312868+0000","last_peered":"2023-07-14T17:42:48.312868+0000","last_clean":"2023-07-14T17:42:48.312868+0000","last_became_active":"2023-07-14T17:42:42.547363+0000","last_became_peered":"2023-07-14T17:42:42.547363+0000","last_unstale":"2023-07-14T17:42:48.312868+0000","last_undegraded":"2023-07-14T17:42:48.312868+0000","last_fullsized":"2023-07-14T17:42:48.312868+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:25:15.031613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T17:43:43.894470+0000","last_change":"2023-07-14T17:42:42.551042+0000","last_active":"2023-07-14T17:43:43.894470+0000","last_peered":"2023-07-14T17:43:43.894470+0000","last_clean":"2023-07-14T17:43:43.894470+0000","last_became_active":"2023-07-14T17:42:42.550730+0000","last_became_peered":"2023-07-14T17:42:42.550730+0000","last_unstale":"2023-07-14T17:43:43.894470+0000","last_undegraded":"2023-07-14T17:43:43.894470+0000","last_fullsized":"2023-07-14T17:43:43.894470+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:03:28.333415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892180+0000","last_change":"2023-07-14T17:42:42.540528+0000","last_active":"2023-07-14T17:43:44.892180+0000","last_peered":"2023-07-14T17:43:44.892180+0000","last_clean":"2023-07-14T17:43:44.892180+0000","last_became_active":"2023-07-14T17:42:42.540093+0000","last_became_peered":"2023-07-14T17:42:42.540093+0000","last_unstale":"2023-07-14T17:43:44.892180+0000","last_undegraded":"2023-07-14T17:43:44.892180+0000","last_fullsized":"2023-07-14T17:43:44.892180+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:13:34.468585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833464+0000","last_change":"2023-07-14T17:42:42.539512+0000","last_active":"2023-07-14T17:42:47.833464+0000","last_peered":"2023-07-14T17:42:47.833464+0000","last_clean":"2023-07-14T17:42:47.833464+0000","last_became_active":"2023-07-14T17:42:42.539248+0000","last_became_peered":"2023-07-14T17:42:42.539248+0000","last_unstale":"2023-07-14T17:42:47.833464+0000","last_undegraded":"2023-07-14T17:42:47.833464+0000","last_fullsized":"2023-07-14T17:42:47.833464+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:02:47.593908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895813+0000","last_change":"2023-07-14T17:42:42.531630+0000","last_active":"2023-07-14T17:43:44.895813+0000","last_peered":"2023-07-14T17:43:44.895813+0000","last_clean":"2023-07-14T17:43:44.895813+0000","last_became_active":"2023-07-14T17:42:42.531333+0000","last_became_peered":"2023-07-14T17:42:42.531333+0000","last_unstale":"2023-07-14T17:43:44.895813+0000","last_undegraded":"2023-07-14T17:43:44.895813+0000","last_fullsized":"2023-07-14T17:43:44.895813+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:37:35.706668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894477+0000","last_change":"2023-07-14T17:42:46.543062+0000","last_active":"2023-07-14T17:43:44.894477+0000","last_peered":"2023-07-14T17:43:44.894477+0000","last_clean":"2023-07-14T17:43:44.894477+0000","last_became_active":"2023-07-14T17:42:46.542189+0000","last_became_peered":"2023-07-14T17:42:46.542189+0000","last_unstale":"2023-07-14T17:43:44.894477+0000","last_undegraded":"2023-07-14T17:43:44.894477+0000","last_fullsized":"2023-07-14T17:43:44.894477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:57:03.881795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":75,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:41.854235+0000","last_change":"2023-07-14T17:42:41.914589+0000","last_active":"2023-07-14T17:43:41.854235+0000","last_peered":"2023-07-14T17:43:41.854235+0000","last_clean":"2023-07-14T17:43:41.854235+0000","last_became_active":"2023-07-14T17:42:41.914275+0000","last_became_peered":"2023-07-14T17:42:41.914275+0000","last_unstale":"2023-07-14T17:43:41.854235+0000","last_undegraded":"2023-07-14T17:43:41.854235+0000","last_fullsized":"2023-07-14T17:43:41.854235+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:09:10.487178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.900792+0000","last_change":"2023-07-14T17:43:44.900792+0000","last_active":"2023-07-14T17:43:44.900792+0000","last_peered":"2023-07-14T17:43:44.900792+0000","last_clean":"2023-07-14T17:43:44.900792+0000","last_became_active":"2023-07-14T17:43:44.900333+0000","last_became_peered":"2023-07-14T17:43:44.900333+0000","last_unstale":"2023-07-14T17:43:44.900792+0000","last_undegraded":"2023-07-14T17:43:44.900792+0000","last_fullsized":"2023-07-14T17:43:44.900792+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:29:20.451913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.541631+0000","last_change":"2023-07-14T17:42:46.542500+0000","last_active":"2023-07-14T17:42:47.541631+0000","last_peered":"2023-07-14T17:42:47.541631+0000","last_clean":"2023-07-14T17:42:47.541631+0000","last_became_active":"2023-07-14T17:42:46.541932+0000","last_became_peered":"2023-07-14T17:42:46.541932+0000","last_unstale":"2023-07-14T17:42:47.541631+0000","last_undegraded":"2023-07-14T17:42:47.541631+0000","last_fullsized":"2023-07-14T17:42:47.541631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:07:53.233373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897058+0000","last_change":"2023-07-14T17:42:41.918169+0000","last_active":"2023-07-14T17:43:44.897058+0000","last_peered":"2023-07-14T17:43:44.897058+0000","last_clean":"2023-07-14T17:43:44.897058+0000","last_became_active":"2023-07-14T17:42:41.917124+0000","last_became_peered":"2023-07-14T17:42:41.917124+0000","last_unstale":"2023-07-14T17:43:44.897058+0000","last_undegraded":"2023-07-14T17:43:44.897058+0000","last_fullsized":"2023-07-14T17:43:44.897058+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:05:43.117229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313826+0000","last_change":"2023-07-14T17:42:42.539265+0000","last_active":"2023-07-14T17:42:48.313826+0000","last_peered":"2023-07-14T17:42:48.313826+0000","last_clean":"2023-07-14T17:42:48.313826+0000","last_became_active":"2023-07-14T17:42:42.538933+0000","last_became_peered":"2023-07-14T17:42:42.538933+0000","last_unstale":"2023-07-14T17:42:48.313826+0000","last_undegraded":"2023-07-14T17:42:48.313826+0000","last_fullsized":"2023-07-14T17:42:48.313826+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:08.277561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542003+0000","last_change":"2023-07-14T17:42:46.550194+0000","last_active":"2023-07-14T17:42:47.542003+0000","last_peered":"2023-07-14T17:42:47.542003+0000","last_clean":"2023-07-14T17:42:47.542003+0000","last_became_active":"2023-07-14T17:42:46.549693+0000","last_became_peered":"2023-07-14T17:42:46.549693+0000","last_unstale":"2023-07-14T17:42:47.542003+0000","last_undegraded":"2023-07-14T17:42:47.542003+0000","last_fullsized":"2023-07-14T17:42:47.542003+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:15.448341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.908529+0000","last_change":"2023-07-14T17:43:44.908253+0000","last_active":"2023-07-14T17:43:44.908529+0000","last_peered":"2023-07-14T17:43:44.908529+0000","last_clean":"2023-07-14T17:43:44.908529+0000","last_became_active":"2023-07-14T17:43:44.896490+0000","last_became_peered":"2023-07-14T17:43:44.896490+0000","last_unstale":"2023-07-14T17:43:44.908529+0000","last_undegraded":"2023-07-14T17:43:44.908529+0000","last_fullsized":"2023-07-14T17:43:44.908529+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:21:55.326452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T17:43:43.894585+0000","last_change":"2023-07-14T17:42:42.543846+0000","last_active":"2023-07-14T17:43:43.894585+0000","last_peered":"2023-07-14T17:43:43.894585+0000","last_clean":"2023-07-14T17:43:43.894585+0000","last_became_active":"2023-07-14T17:42:42.543528+0000","last_became_peered":"2023-07-14T17:42:42.543528+0000","last_unstale":"2023-07-14T17:43:43.894585+0000","last_undegraded":"2023-07-14T17:43:43.894585+0000","last_fullsized":"2023-07-14T17:43:43.894585+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:56:03.460899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.541773+0000","last_change":"2023-07-14T17:42:46.547799+0000","last_active":"2023-07-14T17:42:47.541773+0000","last_peered":"2023-07-14T17:42:47.541773+0000","last_clean":"2023-07-14T17:42:47.541773+0000","last_became_active":"2023-07-14T17:42:46.547381+0000","last_became_peered":"2023-07-14T17:42:46.547381+0000","last_unstale":"2023-07-14T17:42:47.541773+0000","last_undegraded":"2023-07-14T17:42:47.541773+0000","last_fullsized":"2023-07-14T17:42:47.541773+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:46:15.220406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":28,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.771498+0000","last_change":"2023-07-14T17:42:41.914911+0000","last_active":"2023-07-14T17:43:03.771498+0000","last_peered":"2023-07-14T17:43:03.771498+0000","last_clean":"2023-07-14T17:43:03.771498+0000","last_became_active":"2023-07-14T17:42:41.914607+0000","last_became_peered":"2023-07-14T17:42:41.914607+0000","last_unstale":"2023-07-14T17:43:03.771498+0000","last_undegraded":"2023-07-14T17:43:03.771498+0000","last_fullsized":"2023-07-14T17:43:03.771498+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:45.771966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897636+0000","last_change":"2023-07-14T17:42:42.548866+0000","last_active":"2023-07-14T17:43:44.897636+0000","last_peered":"2023-07-14T17:43:44.897636+0000","last_clean":"2023-07-14T17:43:44.897636+0000","last_became_active":"2023-07-14T17:42:42.548562+0000","last_became_peered":"2023-07-14T17:42:42.548562+0000","last_unstale":"2023-07-14T17:43:44.897636+0000","last_undegraded":"2023-07-14T17:43:44.897636+0000","last_fullsized":"2023-07-14T17:43:44.897636+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:41:33.093669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895655+0000","last_change":"2023-07-14T17:42:46.538879+0000","last_active":"2023-07-14T17:43:44.895655+0000","last_peered":"2023-07-14T17:43:44.895655+0000","last_clean":"2023-07-14T17:43:44.895655+0000","last_became_active":"2023-07-14T17:42:46.538476+0000","last_became_peered":"2023-07-14T17:42:46.538476+0000","last_unstale":"2023-07-14T17:43:44.895655+0000","last_undegraded":"2023-07-14T17:43:44.895655+0000","last_fullsized":"2023-07-14T17:43:44.895655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:50:18.995823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":75,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897652+0000","last_change":"2023-07-14T17:42:41.917215+0000","last_active":"2023-07-14T17:43:44.897652+0000","last_peered":"2023-07-14T17:43:44.897652+0000","last_clean":"2023-07-14T17:43:44.897652+0000","last_became_active":"2023-07-14T17:42:41.916640+0000","last_became_peered":"2023-07-14T17:42:41.916640+0000","last_unstale":"2023-07-14T17:43:44.897652+0000","last_undegraded":"2023-07-14T17:43:44.897652+0000","last_fullsized":"2023-07-14T17:43:44.897652+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:57:44.899889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835389+0000","last_change":"2023-07-14T17:42:42.534347+0000","last_active":"2023-07-14T17:42:47.835389+0000","last_peered":"2023-07-14T17:42:47.835389+0000","last_clean":"2023-07-14T17:42:47.835389+0000","last_became_active":"2023-07-14T17:42:42.533958+0000","last_became_peered":"2023-07-14T17:42:42.533958+0000","last_unstale":"2023-07-14T17:42:47.835389+0000","last_undegraded":"2023-07-14T17:42:47.835389+0000","last_fullsized":"2023-07-14T17:42:47.835389+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:52:39.908230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894270+0000","last_change":"2023-07-14T17:42:46.543144+0000","last_active":"2023-07-14T17:43:44.894270+0000","last_peered":"2023-07-14T17:43:44.894270+0000","last_clean":"2023-07-14T17:43:44.894270+0000","last_became_active":"2023-07-14T17:42:46.542311+0000","last_became_peered":"2023-07-14T17:42:46.542311+0000","last_unstale":"2023-07-14T17:43:44.894270+0000","last_undegraded":"2023-07-14T17:43:44.894270+0000","last_fullsized":"2023-07-14T17:43:44.894270+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:21:37.953172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896529+0000","last_change":"2023-07-14T17:42:41.916174+0000","last_active":"2023-07-14T17:43:44.896529+0000","last_peered":"2023-07-14T17:43:44.896529+0000","last_clean":"2023-07-14T17:43:44.896529+0000","last_became_active":"2023-07-14T17:42:41.915867+0000","last_became_peered":"2023-07-14T17:42:41.915867+0000","last_unstale":"2023-07-14T17:43:44.896529+0000","last_undegraded":"2023-07-14T17:43:44.896529+0000","last_fullsized":"2023-07-14T17:43:44.896529+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:59:36.014596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897789+0000","last_change":"2023-07-14T17:42:42.547064+0000","last_active":"2023-07-14T17:43:44.897789+0000","last_peered":"2023-07-14T17:43:44.897789+0000","last_clean":"2023-07-14T17:43:44.897789+0000","last_became_active":"2023-07-14T17:42:42.546777+0000","last_became_peered":"2023-07-14T17:42:42.546777+0000","last_unstale":"2023-07-14T17:43:44.897789+0000","last_undegraded":"2023-07-14T17:43:44.897789+0000","last_fullsized":"2023-07-14T17:43:44.897789+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:44:45.781755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892451+0000","last_change":"2023-07-14T17:42:46.548647+0000","last_active":"2023-07-14T17:43:44.892451+0000","last_peered":"2023-07-14T17:43:44.892451+0000","last_clean":"2023-07-14T17:43:44.892451+0000","last_became_active":"2023-07-14T17:42:46.548142+0000","last_became_peered":"2023-07-14T17:42:46.548142+0000","last_unstale":"2023-07-14T17:43:44.892451+0000","last_undegraded":"2023-07-14T17:43:44.892451+0000","last_fullsized":"2023-07-14T17:43:44.892451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:50:46.460689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324145+0000","last_change":"2023-07-14T17:42:41.911675+0000","last_active":"2023-07-14T17:43:45.324145+0000","last_peered":"2023-07-14T17:43:45.324145+0000","last_clean":"2023-07-14T17:43:45.324145+0000","last_became_active":"2023-07-14T17:42:41.911299+0000","last_became_peered":"2023-07-14T17:42:41.911299+0000","last_unstale":"2023-07-14T17:43:45.324145+0000","last_undegraded":"2023-07-14T17:43:45.324145+0000","last_fullsized":"2023-07-14T17:43:45.324145+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:37:15.112003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895160+0000","last_change":"2023-07-14T17:42:42.536211+0000","last_active":"2023-07-14T17:43:44.895160+0000","last_peered":"2023-07-14T17:43:44.895160+0000","last_clean":"2023-07-14T17:43:44.895160+0000","last_became_active":"2023-07-14T17:42:42.535763+0000","last_became_peered":"2023-07-14T17:42:42.535763+0000","last_unstale":"2023-07-14T17:43:44.895160+0000","last_undegraded":"2023-07-14T17:43:44.895160+0000","last_fullsized":"2023-07-14T17:43:44.895160+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:14:23.840550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313179+0000","last_change":"2023-07-14T17:42:46.540917+0000","last_active":"2023-07-14T17:42:48.313179+0000","last_peered":"2023-07-14T17:42:48.313179+0000","last_clean":"2023-07-14T17:42:48.313179+0000","last_became_active":"2023-07-14T17:42:46.540575+0000","last_became_peered":"2023-07-14T17:42:46.540575+0000","last_unstale":"2023-07-14T17:42:48.313179+0000","last_undegraded":"2023-07-14T17:42:48.313179+0000","last_fullsized":"2023-07-14T17:42:48.313179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:02:20.835521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897945+0000","last_change":"2023-07-14T17:42:41.917646+0000","last_active":"2023-07-14T17:43:44.897945+0000","last_peered":"2023-07-14T17:43:44.897945+0000","last_clean":"2023-07-14T17:43:44.897945+0000","last_became_active":"2023-07-14T17:42:41.917331+0000","last_became_peered":"2023-07-14T17:42:41.917331+0000","last_unstale":"2023-07-14T17:43:44.897945+0000","last_undegraded":"2023-07-14T17:43:44.897945+0000","last_fullsized":"2023-07-14T17:43:44.897945+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:52:16.441394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833094+0000","last_change":"2023-07-14T17:42:42.526696+0000","last_active":"2023-07-14T17:42:47.833094+0000","last_peered":"2023-07-14T17:42:47.833094+0000","last_clean":"2023-07-14T17:42:47.833094+0000","last_became_active":"2023-07-14T17:42:42.526353+0000","last_became_peered":"2023-07-14T17:42:42.526353+0000","last_unstale":"2023-07-14T17:42:47.833094+0000","last_undegraded":"2023-07-14T17:42:47.833094+0000","last_fullsized":"2023-07-14T17:42:47.833094+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:01:54.943614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895396+0000","last_change":"2023-07-14T17:42:46.545460+0000","last_active":"2023-07-14T17:43:44.895396+0000","last_peered":"2023-07-14T17:43:44.895396+0000","last_clean":"2023-07-14T17:43:44.895396+0000","last_became_active":"2023-07-14T17:42:46.544897+0000","last_became_peered":"2023-07-14T17:42:46.544897+0000","last_unstale":"2023-07-14T17:43:44.895396+0000","last_undegraded":"2023-07-14T17:43:44.895396+0000","last_fullsized":"2023-07-14T17:43:44.895396+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:59:41.696162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":112,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896978+0000","last_change":"2023-07-14T17:42:41.912266+0000","last_active":"2023-07-14T17:43:44.896978+0000","last_peered":"2023-07-14T17:43:44.896978+0000","last_clean":"2023-07-14T17:43:44.896978+0000","last_became_active":"2023-07-14T17:42:41.911944+0000","last_became_peered":"2023-07-14T17:42:41.911944+0000","last_unstale":"2023-07-14T17:43:44.896978+0000","last_undegraded":"2023-07-14T17:43:44.896978+0000","last_fullsized":"2023-07-14T17:43:44.896978+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:17:02.788197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":82,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T17:43:43.897418+0000","last_change":"2023-07-14T17:42:42.548609+0000","last_active":"2023-07-14T17:43:43.897418+0000","last_peered":"2023-07-14T17:43:43.897418+0000","last_clean":"2023-07-14T17:43:43.897418+0000","last_became_active":"2023-07-14T17:42:42.548253+0000","last_became_peered":"2023-07-14T17:42:42.548253+0000","last_unstale":"2023-07-14T17:43:43.897418+0000","last_undegraded":"2023-07-14T17:43:43.897418+0000","last_fullsized":"2023-07-14T17:43:43.897418+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:17:37.759289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833114+0000","last_change":"2023-07-14T17:42:46.547324+0000","last_active":"2023-07-14T17:42:47.833114+0000","last_peered":"2023-07-14T17:42:47.833114+0000","last_clean":"2023-07-14T17:42:47.833114+0000","last_became_active":"2023-07-14T17:42:46.546876+0000","last_became_peered":"2023-07-14T17:42:46.546876+0000","last_unstale":"2023-07-14T17:42:47.833114+0000","last_undegraded":"2023-07-14T17:42:47.833114+0000","last_fullsized":"2023-07-14T17:42:47.833114+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:30:13.764106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.818146+0000","last_change":"2023-07-14T17:43:45.818146+0000","last_active":"2023-07-14T17:43:45.818146+0000","last_peered":"2023-07-14T17:43:45.818146+0000","last_clean":"2023-07-14T17:43:45.818146+0000","last_became_active":"2023-07-14T17:43:45.814954+0000","last_became_peered":"2023-07-14T17:43:45.814954+0000","last_unstale":"2023-07-14T17:43:45.818146+0000","last_undegraded":"2023-07-14T17:43:45.818146+0000","last_fullsized":"2023-07-14T17:43:45.818146+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:50:05.262289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313896+0000","last_change":"2023-07-14T17:42:42.548081+0000","last_active":"2023-07-14T17:42:48.313896+0000","last_peered":"2023-07-14T17:42:48.313896+0000","last_clean":"2023-07-14T17:42:48.313896+0000","last_became_active":"2023-07-14T17:42:42.547876+0000","last_became_peered":"2023-07-14T17:42:42.547876+0000","last_unstale":"2023-07-14T17:42:48.313896+0000","last_undegraded":"2023-07-14T17:42:48.313896+0000","last_fullsized":"2023-07-14T17:42:48.313896+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:53:08.262819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542183+0000","last_change":"2023-07-14T17:42:46.550516+0000","last_active":"2023-07-14T17:42:47.542183+0000","last_peered":"2023-07-14T17:42:47.542183+0000","last_clean":"2023-07-14T17:42:47.542183+0000","last_became_active":"2023-07-14T17:42:46.549588+0000","last_became_peered":"2023-07-14T17:42:46.549588+0000","last_unstale":"2023-07-14T17:42:47.542183+0000","last_undegraded":"2023-07-14T17:42:47.542183+0000","last_fullsized":"2023-07-14T17:42:47.542183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:21.155958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897608+0000","last_change":"2023-07-14T17:42:41.917527+0000","last_active":"2023-07-14T17:43:44.897608+0000","last_peered":"2023-07-14T17:43:44.897608+0000","last_clean":"2023-07-14T17:43:44.897608+0000","last_became_active":"2023-07-14T17:42:41.917239+0000","last_became_peered":"2023-07-14T17:42:41.917239+0000","last_unstale":"2023-07-14T17:43:44.897608+0000","last_undegraded":"2023-07-14T17:43:44.897608+0000","last_fullsized":"2023-07-14T17:43:44.897608+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:07:11.390441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T17:43:43.894801+0000","last_change":"2023-07-14T17:42:42.554866+0000","last_active":"2023-07-14T17:43:43.894801+0000","last_peered":"2023-07-14T17:43:43.894801+0000","last_clean":"2023-07-14T17:43:43.894801+0000","last_became_active":"2023-07-14T17:42:42.554585+0000","last_became_peered":"2023-07-14T17:42:42.554585+0000","last_unstale":"2023-07-14T17:43:43.894801+0000","last_undegraded":"2023-07-14T17:43:43.894801+0000","last_fullsized":"2023-07-14T17:43:43.894801+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:56:27.198044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833167+0000","last_change":"2023-07-14T17:42:46.544788+0000","last_active":"2023-07-14T17:42:47.833167+0000","last_peered":"2023-07-14T17:42:47.833167+0000","last_clean":"2023-07-14T17:42:47.833167+0000","last_became_active":"2023-07-14T17:42:46.544419+0000","last_became_peered":"2023-07-14T17:42:46.544419+0000","last_unstale":"2023-07-14T17:42:47.833167+0000","last_undegraded":"2023-07-14T17:42:47.833167+0000","last_fullsized":"2023-07-14T17:42:47.833167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:03:38.940614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"86'33","reported_seq":147,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896947+0000","last_change":"2023-07-14T17:42:41.917069+0000","last_active":"2023-07-14T17:43:44.896947+0000","last_peered":"2023-07-14T17:43:44.896947+0000","last_clean":"2023-07-14T17:43:44.896947+0000","last_became_active":"2023-07-14T17:42:41.916630+0000","last_became_peered":"2023-07-14T17:42:41.916630+0000","last_unstale":"2023-07-14T17:43:44.896947+0000","last_undegraded":"2023-07-14T17:43:44.896947+0000","last_fullsized":"2023-07-14T17:43:44.896947+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:52:18.568627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58551,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":88,"num_read_kb":100,"num_write":35,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835769+0000","last_change":"2023-07-14T17:42:42.542124+0000","last_active":"2023-07-14T17:42:47.835769+0000","last_peered":"2023-07-14T17:42:47.835769+0000","last_clean":"2023-07-14T17:42:47.835769+0000","last_became_active":"2023-07-14T17:42:42.541851+0000","last_became_peered":"2023-07-14T17:42:42.541851+0000","last_unstale":"2023-07-14T17:42:47.835769+0000","last_undegraded":"2023-07-14T17:42:47.835769+0000","last_fullsized":"2023-07-14T17:42:47.835769+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:06.253283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897932+0000","last_change":"2023-07-14T17:42:46.546167+0000","last_active":"2023-07-14T17:43:44.897932+0000","last_peered":"2023-07-14T17:43:44.897932+0000","last_clean":"2023-07-14T17:43:44.897932+0000","last_became_active":"2023-07-14T17:42:46.545768+0000","last_became_peered":"2023-07-14T17:42:46.545768+0000","last_unstale":"2023-07-14T17:43:44.897932+0000","last_undegraded":"2023-07-14T17:43:44.897932+0000","last_fullsized":"2023-07-14T17:43:44.897932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:13:37.105161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894811+0000","last_change":"2023-07-14T17:42:41.918858+0000","last_active":"2023-07-14T17:43:44.894811+0000","last_peered":"2023-07-14T17:43:44.894811+0000","last_clean":"2023-07-14T17:43:44.894811+0000","last_became_active":"2023-07-14T17:42:41.918568+0000","last_became_peered":"2023-07-14T17:42:41.918568+0000","last_unstale":"2023-07-14T17:43:44.894811+0000","last_undegraded":"2023-07-14T17:43:44.894811+0000","last_fullsized":"2023-07-14T17:43:44.894811+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:43:09.338997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324334+0000","last_change":"2023-07-14T17:42:42.542310+0000","last_active":"2023-07-14T17:43:45.324334+0000","last_peered":"2023-07-14T17:43:45.324334+0000","last_clean":"2023-07-14T17:43:45.324334+0000","last_became_active":"2023-07-14T17:42:42.541989+0000","last_became_peered":"2023-07-14T17:42:42.541989+0000","last_unstale":"2023-07-14T17:43:45.324334+0000","last_undegraded":"2023-07-14T17:43:45.324334+0000","last_fullsized":"2023-07-14T17:43:45.324334+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:23:26.638546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.325836+0000","last_change":"2023-07-14T17:43:45.325836+0000","last_active":"2023-07-14T17:43:45.325836+0000","last_peered":"2023-07-14T17:43:45.325836+0000","last_clean":"2023-07-14T17:43:45.325836+0000","last_became_active":"2023-07-14T17:43:45.325330+0000","last_became_peered":"2023-07-14T17:43:45.325330+0000","last_unstale":"2023-07-14T17:43:45.325836+0000","last_undegraded":"2023-07-14T17:43:45.325836+0000","last_fullsized":"2023-07-14T17:43:45.325836+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:34:58.970503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897638+0000","last_change":"2023-07-14T17:42:41.916382+0000","last_active":"2023-07-14T17:43:44.897638+0000","last_peered":"2023-07-14T17:43:44.897638+0000","last_clean":"2023-07-14T17:43:44.897638+0000","last_became_active":"2023-07-14T17:42:41.916082+0000","last_became_peered":"2023-07-14T17:42:41.916082+0000","last_unstale":"2023-07-14T17:43:44.897638+0000","last_undegraded":"2023-07-14T17:43:44.897638+0000","last_fullsized":"2023-07-14T17:43:44.897638+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:28:56.714038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894470+0000","last_change":"2023-07-14T17:42:42.546812+0000","last_active":"2023-07-14T17:43:44.894470+0000","last_peered":"2023-07-14T17:43:44.894470+0000","last_clean":"2023-07-14T17:43:44.894470+0000","last_became_active":"2023-07-14T17:42:42.546517+0000","last_became_peered":"2023-07-14T17:42:42.546517+0000","last_unstale":"2023-07-14T17:43:44.894470+0000","last_undegraded":"2023-07-14T17:43:44.894470+0000","last_fullsized":"2023-07-14T17:43:44.894470+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:01:05.671027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833687+0000","last_change":"2023-07-14T17:42:46.547726+0000","last_active":"2023-07-14T17:42:47.833687+0000","last_peered":"2023-07-14T17:42:47.833687+0000","last_clean":"2023-07-14T17:42:47.833687+0000","last_became_active":"2023-07-14T17:42:46.547207+0000","last_became_peered":"2023-07-14T17:42:46.547207+0000","last_unstale":"2023-07-14T17:42:47.833687+0000","last_undegraded":"2023-07-14T17:42:47.833687+0000","last_fullsized":"2023-07-14T17:42:47.833687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:55:16.418869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"86'6","reported_seq":69,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:41.850813+0000","last_change":"2023-07-14T17:42:41.913802+0000","last_active":"2023-07-14T17:43:41.850813+0000","last_peered":"2023-07-14T17:43:41.850813+0000","last_clean":"2023-07-14T17:43:41.850813+0000","last_became_active":"2023-07-14T17:42:41.913467+0000","last_became_peered":"2023-07-14T17:42:41.913467+0000","last_unstale":"2023-07-14T17:43:41.850813+0000","last_undegraded":"2023-07-14T17:43:41.850813+0000","last_fullsized":"2023-07-14T17:43:41.850813+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:46:59.725226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":42,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894173+0000","last_change":"2023-07-14T17:42:42.545379+0000","last_active":"2023-07-14T17:43:44.894173+0000","last_peered":"2023-07-14T17:43:44.894173+0000","last_clean":"2023-07-14T17:43:44.894173+0000","last_became_active":"2023-07-14T17:42:42.545012+0000","last_became_peered":"2023-07-14T17:42:42.545012+0000","last_unstale":"2023-07-14T17:43:44.894173+0000","last_undegraded":"2023-07-14T17:43:44.894173+0000","last_fullsized":"2023-07-14T17:43:44.894173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:53:26.889969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324438+0000","last_change":"2023-07-14T17:42:46.544524+0000","last_active":"2023-07-14T17:43:45.324438+0000","last_peered":"2023-07-14T17:43:45.324438+0000","last_clean":"2023-07-14T17:43:45.324438+0000","last_became_active":"2023-07-14T17:42:46.544152+0000","last_became_peered":"2023-07-14T17:42:46.544152+0000","last_unstale":"2023-07-14T17:43:45.324438+0000","last_undegraded":"2023-07-14T17:43:45.324438+0000","last_fullsized":"2023-07-14T17:43:45.324438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:15:31.705985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.771510+0000","last_change":"2023-07-14T17:42:41.917282+0000","last_active":"2023-07-14T17:43:03.771510+0000","last_peered":"2023-07-14T17:43:03.771510+0000","last_clean":"2023-07-14T17:43:03.771510+0000","last_became_active":"2023-07-14T17:42:41.916994+0000","last_became_peered":"2023-07-14T17:42:41.916994+0000","last_unstale":"2023-07-14T17:43:03.771510+0000","last_undegraded":"2023-07-14T17:43:03.771510+0000","last_fullsized":"2023-07-14T17:43:03.771510+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:41:27.420237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897640+0000","last_change":"2023-07-14T17:42:42.548525+0000","last_active":"2023-07-14T17:43:44.897640+0000","last_peered":"2023-07-14T17:43:44.897640+0000","last_clean":"2023-07-14T17:43:44.897640+0000","last_became_active":"2023-07-14T17:42:42.548270+0000","last_became_peered":"2023-07-14T17:42:42.548270+0000","last_unstale":"2023-07-14T17:43:44.897640+0000","last_undegraded":"2023-07-14T17:43:44.897640+0000","last_fullsized":"2023-07-14T17:43:44.897640+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:14:31.331118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.898128+0000","last_change":"2023-07-14T17:42:46.546480+0000","last_active":"2023-07-14T17:43:44.898128+0000","last_peered":"2023-07-14T17:43:44.898128+0000","last_clean":"2023-07-14T17:43:44.898128+0000","last_became_active":"2023-07-14T17:42:46.546078+0000","last_became_peered":"2023-07-14T17:42:46.546078+0000","last_unstale":"2023-07-14T17:43:44.898128+0000","last_undegraded":"2023-07-14T17:43:44.898128+0000","last_fullsized":"2023-07-14T17:43:44.898128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:41:01.115708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.781649+0000","last_change":"2023-07-14T17:42:41.914226+0000","last_active":"2023-07-14T17:43:04.781649+0000","last_peered":"2023-07-14T17:43:04.781649+0000","last_clean":"2023-07-14T17:43:04.781649+0000","last_became_active":"2023-07-14T17:42:41.913820+0000","last_became_peered":"2023-07-14T17:42:41.913820+0000","last_unstale":"2023-07-14T17:43:04.781649+0000","last_undegraded":"2023-07-14T17:43:04.781649+0000","last_fullsized":"2023-07-14T17:43:04.781649+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:31:36.040725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892019+0000","last_change":"2023-07-14T17:42:42.543549+0000","last_active":"2023-07-14T17:43:44.892019+0000","last_peered":"2023-07-14T17:43:44.892019+0000","last_clean":"2023-07-14T17:43:44.892019+0000","last_became_active":"2023-07-14T17:42:42.543245+0000","last_became_peered":"2023-07-14T17:42:42.543245+0000","last_unstale":"2023-07-14T17:43:44.892019+0000","last_undegraded":"2023-07-14T17:43:44.892019+0000","last_fullsized":"2023-07-14T17:43:44.892019+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:18:13.987338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835441+0000","last_change":"2023-07-14T17:42:46.577245+0000","last_active":"2023-07-14T17:42:47.835441+0000","last_peered":"2023-07-14T17:42:47.835441+0000","last_clean":"2023-07-14T17:42:47.835441+0000","last_became_active":"2023-07-14T17:42:46.576793+0000","last_became_peered":"2023-07-14T17:42:46.576793+0000","last_unstale":"2023-07-14T17:42:47.835441+0000","last_undegraded":"2023-07-14T17:42:47.835441+0000","last_fullsized":"2023-07-14T17:42:47.835441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:29:32.688546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833389+0000","last_change":"2023-07-14T17:42:41.917199+0000","last_active":"2023-07-14T17:42:47.833389+0000","last_peered":"2023-07-14T17:42:47.833389+0000","last_clean":"2023-07-14T17:42:47.833389+0000","last_became_active":"2023-07-14T17:42:41.916924+0000","last_became_peered":"2023-07-14T17:42:41.916924+0000","last_unstale":"2023-07-14T17:42:47.833389+0000","last_undegraded":"2023-07-14T17:42:47.833389+0000","last_fullsized":"2023-07-14T17:42:47.833389+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:12:02.515835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896869+0000","last_change":"2023-07-14T17:42:42.540531+0000","last_active":"2023-07-14T17:43:44.896869+0000","last_peered":"2023-07-14T17:43:44.896869+0000","last_clean":"2023-07-14T17:43:44.896869+0000","last_became_active":"2023-07-14T17:42:42.540165+0000","last_became_peered":"2023-07-14T17:42:42.540165+0000","last_unstale":"2023-07-14T17:43:44.896869+0000","last_undegraded":"2023-07-14T17:43:44.896869+0000","last_fullsized":"2023-07-14T17:43:44.896869+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:30:36.053357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324213+0000","last_change":"2023-07-14T17:42:46.537158+0000","last_active":"2023-07-14T17:43:45.324213+0000","last_peered":"2023-07-14T17:43:45.324213+0000","last_clean":"2023-07-14T17:43:45.324213+0000","last_became_active":"2023-07-14T17:42:46.536816+0000","last_became_peered":"2023-07-14T17:42:46.536816+0000","last_unstale":"2023-07-14T17:43:45.324213+0000","last_undegraded":"2023-07-14T17:43:45.324213+0000","last_fullsized":"2023-07-14T17:43:45.324213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:00:28.047931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894268+0000","last_change":"2023-07-14T17:42:41.913638+0000","last_active":"2023-07-14T17:43:44.894268+0000","last_peered":"2023-07-14T17:43:44.894268+0000","last_clean":"2023-07-14T17:43:44.894268+0000","last_became_active":"2023-07-14T17:42:41.913361+0000","last_became_peered":"2023-07-14T17:42:41.913361+0000","last_unstale":"2023-07-14T17:43:44.894268+0000","last_undegraded":"2023-07-14T17:43:44.894268+0000","last_fullsized":"2023-07-14T17:43:44.894268+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:46:04.106529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.901414+0000","last_change":"2023-07-14T17:43:44.901414+0000","last_active":"2023-07-14T17:43:44.901414+0000","last_peered":"2023-07-14T17:43:44.901414+0000","last_clean":"2023-07-14T17:43:44.901414+0000","last_became_active":"2023-07-14T17:43:44.901164+0000","last_became_peered":"2023-07-14T17:43:44.901164+0000","last_unstale":"2023-07-14T17:43:44.901414+0000","last_undegraded":"2023-07-14T17:43:44.901414+0000","last_fullsized":"2023-07-14T17:43:44.901414+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:15:25.526757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324220+0000","last_change":"2023-07-14T17:42:46.576999+0000","last_active":"2023-07-14T17:43:45.324220+0000","last_peered":"2023-07-14T17:43:45.324220+0000","last_clean":"2023-07-14T17:43:45.324220+0000","last_became_active":"2023-07-14T17:42:46.576652+0000","last_became_peered":"2023-07-14T17:42:46.576652+0000","last_unstale":"2023-07-14T17:43:45.324220+0000","last_undegraded":"2023-07-14T17:43:45.324220+0000","last_fullsized":"2023-07-14T17:43:45.324220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:14.124797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897667+0000","last_change":"2023-07-14T17:42:42.547053+0000","last_active":"2023-07-14T17:43:44.897667+0000","last_peered":"2023-07-14T17:43:44.897667+0000","last_clean":"2023-07-14T17:43:44.897667+0000","last_became_active":"2023-07-14T17:42:42.546735+0000","last_became_peered":"2023-07-14T17:42:42.546735+0000","last_unstale":"2023-07-14T17:43:44.897667+0000","last_undegraded":"2023-07-14T17:43:44.897667+0000","last_fullsized":"2023-07-14T17:43:44.897667+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:48:04.951463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324057+0000","last_change":"2023-07-14T17:42:41.918692+0000","last_active":"2023-07-14T17:43:45.324057+0000","last_peered":"2023-07-14T17:43:45.324057+0000","last_clean":"2023-07-14T17:43:45.324057+0000","last_became_active":"2023-07-14T17:42:41.918360+0000","last_became_peered":"2023-07-14T17:42:41.918360+0000","last_unstale":"2023-07-14T17:43:45.324057+0000","last_undegraded":"2023-07-14T17:43:45.324057+0000","last_fullsized":"2023-07-14T17:43:45.324057+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:13:06.540310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894327+0000","last_change":"2023-07-14T17:42:46.543997+0000","last_active":"2023-07-14T17:43:44.894327+0000","last_peered":"2023-07-14T17:43:44.894327+0000","last_clean":"2023-07-14T17:43:44.894327+0000","last_became_active":"2023-07-14T17:42:46.543650+0000","last_became_peered":"2023-07-14T17:42:46.543650+0000","last_unstale":"2023-07-14T17:43:44.894327+0000","last_undegraded":"2023-07-14T17:43:44.894327+0000","last_fullsized":"2023-07-14T17:43:44.894327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:30:06.964324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895872+0000","last_change":"2023-07-14T17:42:42.548860+0000","last_active":"2023-07-14T17:43:44.895872+0000","last_peered":"2023-07-14T17:43:44.895872+0000","last_clean":"2023-07-14T17:43:44.895872+0000","last_became_active":"2023-07-14T17:42:42.548564+0000","last_became_peered":"2023-07-14T17:42:42.548564+0000","last_unstale":"2023-07-14T17:43:44.895872+0000","last_undegraded":"2023-07-14T17:43:44.895872+0000","last_fullsized":"2023-07-14T17:43:44.895872+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:51:19.885558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:04.782501+0000","last_change":"2023-07-14T17:42:41.912601+0000","last_active":"2023-07-14T17:43:04.782501+0000","last_peered":"2023-07-14T17:43:04.782501+0000","last_clean":"2023-07-14T17:43:04.782501+0000","last_became_active":"2023-07-14T17:42:41.912265+0000","last_became_peered":"2023-07-14T17:42:41.912265+0000","last_unstale":"2023-07-14T17:43:04.782501+0000","last_undegraded":"2023-07-14T17:43:04.782501+0000","last_fullsized":"2023-07-14T17:43:04.782501+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:51:58.872606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":347,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894007+0000","last_change":"2023-07-14T17:41:43.696093+0000","last_active":"2023-07-14T17:43:44.894007+0000","last_peered":"2023-07-14T17:43:44.894007+0000","last_clean":"2023-07-14T17:43:44.894007+0000","last_became_active":"2023-07-14T17:41:43.695511+0000","last_became_peered":"2023-07-14T17:41:43.695511+0000","last_unstale":"2023-07-14T17:43:44.894007+0000","last_undegraded":"2023-07-14T17:43:44.894007+0000","last_fullsized":"2023-07-14T17:43:44.894007+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:38:50.657865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:38:50.657865+0000","last_clean_scrub_stamp":"2023-07-14T17:38:50.657865+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:01:26.054999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"activating","last_fresh":"2023-07-14T17:43:44.900723+0000","last_change":"2023-07-14T17:43:44.897633+0000","last_active":"2023-07-14T17:43:43.899017+0000","last_peered":"2023-07-14T17:42:50.079160+0000","last_clean":"2023-07-14T17:42:50.079160+0000","last_became_active":"2023-07-14T17:42:46.577067+0000","last_became_peered":"2023-07-14T17:42:46.577067+0000","last_unstale":"2023-07-14T17:43:44.900723+0000","last_undegraded":"2023-07-14T17:43:44.900723+0000","last_fullsized":"2023-07-14T17:43:44.900723+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:06:30.535779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":["5(0)","6(2)","9(1)","10(3)","11(1)"],"object_location_counts":[],"blocked_by":[2],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.323905+0000","last_change":"2023-07-14T17:42:42.543243+0000","last_active":"2023-07-14T17:43:45.323905+0000","last_peered":"2023-07-14T17:43:45.323905+0000","last_clean":"2023-07-14T17:43:45.323905+0000","last_became_active":"2023-07-14T17:42:42.542965+0000","last_became_peered":"2023-07-14T17:42:42.542965+0000","last_unstale":"2023-07-14T17:43:45.323905+0000","last_undegraded":"2023-07-14T17:43:45.323905+0000","last_fullsized":"2023-07-14T17:43:45.323905+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:11:31.220902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894165+0000","last_change":"2023-07-14T17:42:41.920806+0000","last_active":"2023-07-14T17:43:44.894165+0000","last_peered":"2023-07-14T17:43:44.894165+0000","last_clean":"2023-07-14T17:43:44.894165+0000","last_became_active":"2023-07-14T17:42:41.920528+0000","last_became_peered":"2023-07-14T17:42:41.920528+0000","last_unstale":"2023-07-14T17:43:44.894165+0000","last_undegraded":"2023-07-14T17:43:44.894165+0000","last_fullsized":"2023-07-14T17:43:44.894165+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:56:57.810566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.891910+0000","last_change":"2023-07-14T17:42:46.542026+0000","last_active":"2023-07-14T17:43:44.891910+0000","last_peered":"2023-07-14T17:43:44.891910+0000","last_clean":"2023-07-14T17:43:44.891910+0000","last_became_active":"2023-07-14T17:42:46.541672+0000","last_became_peered":"2023-07-14T17:42:46.541672+0000","last_unstale":"2023-07-14T17:43:44.891910+0000","last_undegraded":"2023-07-14T17:43:44.891910+0000","last_fullsized":"2023-07-14T17:43:44.891910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:31:32.137890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.772096+0000","last_change":"2023-07-14T17:42:41.915798+0000","last_active":"2023-07-14T17:43:03.772096+0000","last_peered":"2023-07-14T17:43:03.772096+0000","last_clean":"2023-07-14T17:43:03.772096+0000","last_became_active":"2023-07-14T17:42:41.915492+0000","last_became_peered":"2023-07-14T17:42:41.915492+0000","last_unstale":"2023-07-14T17:43:03.772096+0000","last_undegraded":"2023-07-14T17:43:03.772096+0000","last_fullsized":"2023-07-14T17:43:03.772096+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:25:42.031839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324401+0000","last_change":"2023-07-14T17:42:42.533500+0000","last_active":"2023-07-14T17:43:45.324401+0000","last_peered":"2023-07-14T17:43:45.324401+0000","last_clean":"2023-07-14T17:43:45.324401+0000","last_became_active":"2023-07-14T17:42:42.533232+0000","last_became_peered":"2023-07-14T17:42:42.533232+0000","last_unstale":"2023-07-14T17:43:45.324401+0000","last_undegraded":"2023-07-14T17:43:45.324401+0000","last_fullsized":"2023-07-14T17:43:45.324401+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:42:49.105557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895003+0000","last_change":"2023-07-14T17:42:46.577131+0000","last_active":"2023-07-14T17:43:44.895003+0000","last_peered":"2023-07-14T17:43:44.895003+0000","last_clean":"2023-07-14T17:43:44.895003+0000","last_became_active":"2023-07-14T17:42:46.576695+0000","last_became_peered":"2023-07-14T17:42:46.576695+0000","last_unstale":"2023-07-14T17:43:44.895003+0000","last_undegraded":"2023-07-14T17:43:44.895003+0000","last_fullsized":"2023-07-14T17:43:44.895003+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:46:57.153931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'9","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:07.833208+0000","last_change":"2023-07-14T17:42:41.919021+0000","last_active":"2023-07-14T17:43:07.833208+0000","last_peered":"2023-07-14T17:43:07.833208+0000","last_clean":"2023-07-14T17:43:07.833208+0000","last_became_active":"2023-07-14T17:42:41.918714+0000","last_became_peered":"2023-07-14T17:42:41.918714+0000","last_unstale":"2023-07-14T17:43:07.833208+0000","last_undegraded":"2023-07-14T17:43:07.833208+0000","last_fullsized":"2023-07-14T17:43:07.833208+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:40:48.400679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897256+0000","last_change":"2023-07-14T17:42:42.534661+0000","last_active":"2023-07-14T17:43:44.897256+0000","last_peered":"2023-07-14T17:43:44.897256+0000","last_clean":"2023-07-14T17:43:44.897256+0000","last_became_active":"2023-07-14T17:42:42.534357+0000","last_became_peered":"2023-07-14T17:42:42.534357+0000","last_unstale":"2023-07-14T17:43:44.897256+0000","last_undegraded":"2023-07-14T17:43:44.897256+0000","last_fullsized":"2023-07-14T17:43:44.897256+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:13:04.290337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894357+0000","last_change":"2023-07-14T17:42:46.538648+0000","last_active":"2023-07-14T17:43:44.894357+0000","last_peered":"2023-07-14T17:43:44.894357+0000","last_clean":"2023-07-14T17:43:44.894357+0000","last_became_active":"2023-07-14T17:42:46.538225+0000","last_became_peered":"2023-07-14T17:42:46.538225+0000","last_unstale":"2023-07-14T17:43:44.894357+0000","last_undegraded":"2023-07-14T17:43:44.894357+0000","last_fullsized":"2023-07-14T17:43:44.894357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:21:00.195671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":30,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:03.772287+0000","last_change":"2023-07-14T17:42:41.915406+0000","last_active":"2023-07-14T17:43:03.772287+0000","last_peered":"2023-07-14T17:43:03.772287+0000","last_clean":"2023-07-14T17:43:03.772287+0000","last_became_active":"2023-07-14T17:42:41.914967+0000","last_became_peered":"2023-07-14T17:42:41.914967+0000","last_unstale":"2023-07-14T17:43:03.772287+0000","last_undegraded":"2023-07-14T17:43:03.772287+0000","last_fullsized":"2023-07-14T17:43:03.772287+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:56:07.113627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313341+0000","last_change":"2023-07-14T17:42:42.539052+0000","last_active":"2023-07-14T17:42:48.313341+0000","last_peered":"2023-07-14T17:42:48.313341+0000","last_clean":"2023-07-14T17:42:48.313341+0000","last_became_active":"2023-07-14T17:42:42.538752+0000","last_became_peered":"2023-07-14T17:42:42.538752+0000","last_unstale":"2023-07-14T17:42:48.313341+0000","last_undegraded":"2023-07-14T17:42:48.313341+0000","last_fullsized":"2023-07-14T17:42:48.313341+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:25:31.544906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.542387+0000","last_change":"2023-07-14T17:42:46.577224+0000","last_active":"2023-07-14T17:42:47.542387+0000","last_peered":"2023-07-14T17:42:47.542387+0000","last_clean":"2023-07-14T17:42:47.542387+0000","last_became_active":"2023-07-14T17:42:46.576781+0000","last_became_peered":"2023-07-14T17:42:46.576781+0000","last_unstale":"2023-07-14T17:42:47.542387+0000","last_undegraded":"2023-07-14T17:42:47.542387+0000","last_fullsized":"2023-07-14T17:42:47.542387+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:47:23.639743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T17:43:43.894174+0000","last_change":"2023-07-14T17:42:41.916406+0000","last_active":"2023-07-14T17:43:43.894174+0000","last_peered":"2023-07-14T17:43:43.894174+0000","last_clean":"2023-07-14T17:43:43.894174+0000","last_became_active":"2023-07-14T17:42:41.916115+0000","last_became_peered":"2023-07-14T17:42:41.916115+0000","last_unstale":"2023-07-14T17:43:43.894174+0000","last_undegraded":"2023-07-14T17:43:43.894174+0000","last_fullsized":"2023-07-14T17:43:43.894174+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:24:40.609115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.835280+0000","last_change":"2023-07-14T17:42:42.529969+0000","last_active":"2023-07-14T17:42:47.835280+0000","last_peered":"2023-07-14T17:42:47.835280+0000","last_clean":"2023-07-14T17:42:47.835280+0000","last_became_active":"2023-07-14T17:42:42.529659+0000","last_became_peered":"2023-07-14T17:42:42.529659+0000","last_unstale":"2023-07-14T17:42:47.835280+0000","last_undegraded":"2023-07-14T17:42:47.835280+0000","last_fullsized":"2023-07-14T17:42:47.835280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:45:58.093667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.833742+0000","last_change":"2023-07-14T17:42:46.546303+0000","last_active":"2023-07-14T17:42:47.833742+0000","last_peered":"2023-07-14T17:42:47.833742+0000","last_clean":"2023-07-14T17:42:47.833742+0000","last_became_active":"2023-07-14T17:42:46.545947+0000","last_became_peered":"2023-07-14T17:42:46.545947+0000","last_unstale":"2023-07-14T17:42:47.833742+0000","last_undegraded":"2023-07-14T17:42:47.833742+0000","last_fullsized":"2023-07-14T17:42:47.833742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:08:49.113221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895642+0000","last_change":"2023-07-14T17:42:41.916373+0000","last_active":"2023-07-14T17:43:44.895642+0000","last_peered":"2023-07-14T17:43:44.895642+0000","last_clean":"2023-07-14T17:43:44.895642+0000","last_became_active":"2023-07-14T17:42:41.915966+0000","last_became_peered":"2023-07-14T17:42:41.915966+0000","last_unstale":"2023-07-14T17:43:44.895642+0000","last_undegraded":"2023-07-14T17:43:44.895642+0000","last_fullsized":"2023-07-14T17:43:44.895642+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:32:53.280585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.891773+0000","last_change":"2023-07-14T17:42:42.532040+0000","last_active":"2023-07-14T17:43:44.891773+0000","last_peered":"2023-07-14T17:43:44.891773+0000","last_clean":"2023-07-14T17:43:44.891773+0000","last_became_active":"2023-07-14T17:42:42.531690+0000","last_became_peered":"2023-07-14T17:42:42.531690+0000","last_unstale":"2023-07-14T17:43:44.891773+0000","last_undegraded":"2023-07-14T17:43:44.891773+0000","last_fullsized":"2023-07-14T17:43:44.891773+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:19:44.676745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.319751+0000","last_change":"2023-07-14T17:42:46.550166+0000","last_active":"2023-07-14T17:42:48.319751+0000","last_peered":"2023-07-14T17:42:48.319751+0000","last_clean":"2023-07-14T17:42:48.319751+0000","last_became_active":"2023-07-14T17:42:46.549700+0000","last_became_peered":"2023-07-14T17:42:46.549700+0000","last_unstale":"2023-07-14T17:42:48.319751+0000","last_undegraded":"2023-07-14T17:42:48.319751+0000","last_fullsized":"2023-07-14T17:42:48.319751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:23.561271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.911523+0000","last_change":"2023-07-14T17:43:44.911267+0000","last_active":"2023-07-14T17:43:44.911523+0000","last_peered":"2023-07-14T17:43:44.911523+0000","last_clean":"2023-07-14T17:43:44.911523+0000","last_became_active":"2023-07-14T17:43:44.899552+0000","last_became_peered":"2023-07-14T17:43:44.899552+0000","last_unstale":"2023-07-14T17:43:44.911523+0000","last_undegraded":"2023-07-14T17:43:44.911523+0000","last_fullsized":"2023-07-14T17:43:44.911523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:30:32.991380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897748+0000","last_change":"2023-07-14T17:42:42.531789+0000","last_active":"2023-07-14T17:43:44.897748+0000","last_peered":"2023-07-14T17:43:44.897748+0000","last_clean":"2023-07-14T17:43:44.897748+0000","last_became_active":"2023-07-14T17:42:42.531460+0000","last_became_peered":"2023-07-14T17:42:42.531460+0000","last_unstale":"2023-07-14T17:43:44.897748+0000","last_undegraded":"2023-07-14T17:43:44.897748+0000","last_fullsized":"2023-07-14T17:43:44.897748+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:46.112701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:47.541998+0000","last_change":"2023-07-14T17:42:46.577034+0000","last_active":"2023-07-14T17:42:47.541998+0000","last_peered":"2023-07-14T17:42:47.541998+0000","last_clean":"2023-07-14T17:42:47.541998+0000","last_became_active":"2023-07-14T17:42:46.576636+0000","last_became_peered":"2023-07-14T17:42:46.576636+0000","last_unstale":"2023-07-14T17:42:47.541998+0000","last_undegraded":"2023-07-14T17:42:47.541998+0000","last_fullsized":"2023-07-14T17:42:47.541998+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:08:40.397509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:43:00.743345+0000","last_change":"2023-07-14T17:42:41.919099+0000","last_active":"2023-07-14T17:43:00.743345+0000","last_peered":"2023-07-14T17:43:00.743345+0000","last_clean":"2023-07-14T17:43:00.743345+0000","last_became_active":"2023-07-14T17:42:41.918306+0000","last_became_peered":"2023-07-14T17:42:41.918306+0000","last_unstale":"2023-07-14T17:43:00.743345+0000","last_undegraded":"2023-07-14T17:43:00.743345+0000","last_fullsized":"2023-07-14T17:43:00.743345+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:20:09.398489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894358+0000","last_change":"2023-07-14T17:42:42.552969+0000","last_active":"2023-07-14T17:43:44.894358+0000","last_peered":"2023-07-14T17:43:44.894358+0000","last_clean":"2023-07-14T17:43:44.894358+0000","last_became_active":"2023-07-14T17:42:42.552647+0000","last_became_peered":"2023-07-14T17:42:42.552647+0000","last_unstale":"2023-07-14T17:43:44.894358+0000","last_undegraded":"2023-07-14T17:43:44.894358+0000","last_fullsized":"2023-07-14T17:43:44.894358+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:41:32.606687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":15,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T17:43:43.894717+0000","last_change":"2023-07-14T17:42:46.538335+0000","last_active":"2023-07-14T17:43:43.894717+0000","last_peered":"2023-07-14T17:43:43.894717+0000","last_clean":"2023-07-14T17:43:43.894717+0000","last_became_active":"2023-07-14T17:42:46.537929+0000","last_became_peered":"2023-07-14T17:42:46.537929+0000","last_unstale":"2023-07-14T17:43:43.894717+0000","last_undegraded":"2023-07-14T17:43:43.894717+0000","last_fullsized":"2023-07-14T17:43:43.894717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:44:37.297601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":72,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894655+0000","last_change":"2023-07-14T17:42:41.916323+0000","last_active":"2023-07-14T17:43:44.894655+0000","last_peered":"2023-07-14T17:43:44.894655+0000","last_clean":"2023-07-14T17:43:44.894655+0000","last_became_active":"2023-07-14T17:42:41.916019+0000","last_became_peered":"2023-07-14T17:42:41.916019+0000","last_unstale":"2023-07-14T17:43:44.894655+0000","last_undegraded":"2023-07-14T17:43:44.894655+0000","last_fullsized":"2023-07-14T17:43:44.894655+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:29:56.282750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894518+0000","last_change":"2023-07-14T17:42:42.549843+0000","last_active":"2023-07-14T17:43:44.894518+0000","last_peered":"2023-07-14T17:43:44.894518+0000","last_clean":"2023-07-14T17:43:44.894518+0000","last_became_active":"2023-07-14T17:42:42.549347+0000","last_became_peered":"2023-07-14T17:42:42.549347+0000","last_unstale":"2023-07-14T17:43:44.894518+0000","last_undegraded":"2023-07-14T17:43:44.894518+0000","last_fullsized":"2023-07-14T17:43:44.894518+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:32:23.560638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313679+0000","last_change":"2023-07-14T17:42:46.545262+0000","last_active":"2023-07-14T17:42:48.313679+0000","last_peered":"2023-07-14T17:42:48.313679+0000","last_clean":"2023-07-14T17:42:48.313679+0000","last_became_active":"2023-07-14T17:42:46.544891+0000","last_became_peered":"2023-07-14T17:42:46.544891+0000","last_unstale":"2023-07-14T17:42:48.313679+0000","last_undegraded":"2023-07-14T17:42:48.313679+0000","last_fullsized":"2023-07-14T17:42:48.313679+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:58:10.135808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897783+0000","last_change":"2023-07-14T17:42:41.912855+0000","last_active":"2023-07-14T17:43:44.897783+0000","last_peered":"2023-07-14T17:43:44.897783+0000","last_clean":"2023-07-14T17:43:44.897783+0000","last_became_active":"2023-07-14T17:42:41.912510+0000","last_became_peered":"2023-07-14T17:42:41.912510+0000","last_unstale":"2023-07-14T17:43:44.897783+0000","last_undegraded":"2023-07-14T17:43:44.897783+0000","last_fullsized":"2023-07-14T17:43:44.897783+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:29:49.003962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894208+0000","last_change":"2023-07-14T17:42:42.545344+0000","last_active":"2023-07-14T17:43:44.894208+0000","last_peered":"2023-07-14T17:43:44.894208+0000","last_clean":"2023-07-14T17:43:44.894208+0000","last_became_active":"2023-07-14T17:42:42.545056+0000","last_became_peered":"2023-07-14T17:42:42.545056+0000","last_unstale":"2023-07-14T17:43:44.894208+0000","last_undegraded":"2023-07-14T17:43:44.894208+0000","last_fullsized":"2023-07-14T17:43:44.894208+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:31:59.187718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897362+0000","last_change":"2023-07-14T17:42:46.541279+0000","last_active":"2023-07-14T17:43:44.897362+0000","last_peered":"2023-07-14T17:43:44.897362+0000","last_clean":"2023-07-14T17:43:44.897362+0000","last_became_active":"2023-07-14T17:42:46.540835+0000","last_became_peered":"2023-07-14T17:42:46.540835+0000","last_unstale":"2023-07-14T17:43:44.897362+0000","last_undegraded":"2023-07-14T17:43:44.897362+0000","last_fullsized":"2023-07-14T17:43:44.897362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:41:42.325967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324578+0000","last_change":"2023-07-14T17:42:41.919257+0000","last_active":"2023-07-14T17:43:45.324578+0000","last_peered":"2023-07-14T17:43:45.324578+0000","last_clean":"2023-07-14T17:43:45.324578+0000","last_became_active":"2023-07-14T17:42:41.918968+0000","last_became_peered":"2023-07-14T17:42:41.918968+0000","last_unstale":"2023-07-14T17:43:45.324578+0000","last_undegraded":"2023-07-14T17:43:45.324578+0000","last_fullsized":"2023-07-14T17:43:45.324578+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:06:24.227953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894456+0000","last_change":"2023-07-14T17:42:42.543993+0000","last_active":"2023-07-14T17:43:44.894456+0000","last_peered":"2023-07-14T17:43:44.894456+0000","last_clean":"2023-07-14T17:43:44.894456+0000","last_became_active":"2023-07-14T17:42:42.543685+0000","last_became_peered":"2023-07-14T17:42:42.543685+0000","last_unstale":"2023-07-14T17:43:44.894456+0000","last_undegraded":"2023-07-14T17:43:44.894456+0000","last_fullsized":"2023-07-14T17:43:44.894456+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:45:26.338845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.891847+0000","last_change":"2023-07-14T17:42:42.538553+0000","last_active":"2023-07-14T17:43:44.891847+0000","last_peered":"2023-07-14T17:43:44.891847+0000","last_clean":"2023-07-14T17:43:44.891847+0000","last_became_active":"2023-07-14T17:42:42.538251+0000","last_became_peered":"2023-07-14T17:42:42.538251+0000","last_unstale":"2023-07-14T17:43:44.891847+0000","last_undegraded":"2023-07-14T17:43:44.891847+0000","last_fullsized":"2023-07-14T17:43:44.891847+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:44:50.672379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894621+0000","last_change":"2023-07-14T17:42:42.543819+0000","last_active":"2023-07-14T17:43:44.894621+0000","last_peered":"2023-07-14T17:43:44.894621+0000","last_clean":"2023-07-14T17:43:44.894621+0000","last_became_active":"2023-07-14T17:42:42.543311+0000","last_became_peered":"2023-07-14T17:42:42.543311+0000","last_unstale":"2023-07-14T17:43:44.894621+0000","last_undegraded":"2023-07-14T17:43:44.894621+0000","last_fullsized":"2023-07-14T17:43:44.894621+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:03:11.132113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894009+0000","last_change":"2023-07-14T17:42:42.537525+0000","last_active":"2023-07-14T17:43:44.894009+0000","last_peered":"2023-07-14T17:43:44.894009+0000","last_clean":"2023-07-14T17:43:44.894009+0000","last_became_active":"2023-07-14T17:42:42.537242+0000","last_became_peered":"2023-07-14T17:42:42.537242+0000","last_unstale":"2023-07-14T17:43:44.894009+0000","last_undegraded":"2023-07-14T17:43:44.894009+0000","last_fullsized":"2023-07-14T17:43:44.894009+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:40:58.077332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T17:42:48.313834+0000","last_change":"2023-07-14T17:42:42.547670+0000","last_active":"2023-07-14T17:42:48.313834+0000","last_peered":"2023-07-14T17:42:48.313834+0000","last_clean":"2023-07-14T17:42:48.313834+0000","last_became_active":"2023-07-14T17:42:42.547349+0000","last_became_peered":"2023-07-14T17:42:42.547349+0000","last_unstale":"2023-07-14T17:42:48.313834+0000","last_undegraded":"2023-07-14T17:42:48.313834+0000","last_fullsized":"2023-07-14T17:42:48.313834+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:11:40.630882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324379+0000","last_change":"2023-07-14T17:42:42.542741+0000","last_active":"2023-07-14T17:43:45.324379+0000","last_peered":"2023-07-14T17:43:45.324379+0000","last_clean":"2023-07-14T17:43:45.324379+0000","last_became_active":"2023-07-14T17:42:42.541972+0000","last_became_peered":"2023-07-14T17:42:42.541972+0000","last_unstale":"2023-07-14T17:43:45.324379+0000","last_undegraded":"2023-07-14T17:43:45.324379+0000","last_fullsized":"2023-07-14T17:43:45.324379+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:27.843013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892205+0000","last_change":"2023-07-14T17:42:42.535043+0000","last_active":"2023-07-14T17:43:44.892205+0000","last_peered":"2023-07-14T17:43:44.892205+0000","last_clean":"2023-07-14T17:43:44.892205+0000","last_became_active":"2023-07-14T17:42:42.534281+0000","last_became_peered":"2023-07-14T17:42:42.534281+0000","last_unstale":"2023-07-14T17:43:44.892205+0000","last_undegraded":"2023-07-14T17:43:44.892205+0000","last_fullsized":"2023-07-14T17:43:44.892205+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:27:45.984466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895460+0000","last_change":"2023-07-14T17:42:42.540866+0000","last_active":"2023-07-14T17:43:44.895460+0000","last_peered":"2023-07-14T17:43:44.895460+0000","last_clean":"2023-07-14T17:43:44.895460+0000","last_became_active":"2023-07-14T17:42:42.540568+0000","last_became_peered":"2023-07-14T17:42:42.540568+0000","last_unstale":"2023-07-14T17:43:44.895460+0000","last_undegraded":"2023-07-14T17:43:44.895460+0000","last_fullsized":"2023-07-14T17:43:44.895460+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:12:01.698385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897437+0000","last_change":"2023-07-14T17:42:42.541646+0000","last_active":"2023-07-14T17:43:44.897437+0000","last_peered":"2023-07-14T17:43:44.897437+0000","last_clean":"2023-07-14T17:43:44.897437+0000","last_became_active":"2023-07-14T17:42:42.541324+0000","last_became_peered":"2023-07-14T17:42:42.541324+0000","last_unstale":"2023-07-14T17:43:44.897437+0000","last_undegraded":"2023-07-14T17:43:44.897437+0000","last_fullsized":"2023-07-14T17:43:44.897437+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:37:25.561305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897528+0000","last_change":"2023-07-14T17:42:42.534650+0000","last_active":"2023-07-14T17:43:44.897528+0000","last_peered":"2023-07-14T17:43:44.897528+0000","last_clean":"2023-07-14T17:43:44.897528+0000","last_became_active":"2023-07-14T17:42:42.534351+0000","last_became_peered":"2023-07-14T17:42:42.534351+0000","last_unstale":"2023-07-14T17:43:44.897528+0000","last_undegraded":"2023-07-14T17:43:44.897528+0000","last_fullsized":"2023-07-14T17:43:44.897528+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:13:52.397350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895517+0000","last_change":"2023-07-14T17:42:42.535815+0000","last_active":"2023-07-14T17:43:44.895517+0000","last_peered":"2023-07-14T17:43:44.895517+0000","last_clean":"2023-07-14T17:43:44.895517+0000","last_became_active":"2023-07-14T17:42:42.535510+0000","last_became_peered":"2023-07-14T17:42:42.535510+0000","last_unstale":"2023-07-14T17:43:44.895517+0000","last_undegraded":"2023-07-14T17:43:44.895517+0000","last_fullsized":"2023-07-14T17:43:44.895517+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:52:40.052652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T17:43:43.895262+0000","last_change":"2023-07-14T17:42:42.551758+0000","last_active":"2023-07-14T17:43:43.895262+0000","last_peered":"2023-07-14T17:43:43.895262+0000","last_clean":"2023-07-14T17:43:43.895262+0000","last_became_active":"2023-07-14T17:42:42.551444+0000","last_became_peered":"2023-07-14T17:42:42.551444+0000","last_unstale":"2023-07-14T17:43:43.895262+0000","last_undegraded":"2023-07-14T17:43:43.895262+0000","last_fullsized":"2023-07-14T17:43:43.895262+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:52:15.412415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324115+0000","last_change":"2023-07-14T17:42:42.546355+0000","last_active":"2023-07-14T17:43:45.324115+0000","last_peered":"2023-07-14T17:43:45.324115+0000","last_clean":"2023-07-14T17:43:45.324115+0000","last_became_active":"2023-07-14T17:42:42.546079+0000","last_became_peered":"2023-07-14T17:42:42.546079+0000","last_unstale":"2023-07-14T17:43:45.324115+0000","last_undegraded":"2023-07-14T17:43:45.324115+0000","last_fullsized":"2023-07-14T17:43:45.324115+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:57:16.604025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.901014+0000","last_change":"2023-07-14T17:43:44.901014+0000","last_active":"2023-07-14T17:43:44.901014+0000","last_peered":"2023-07-14T17:43:44.901014+0000","last_clean":"2023-07-14T17:43:44.901014+0000","last_became_active":"2023-07-14T17:43:44.900632+0000","last_became_peered":"2023-07-14T17:43:44.900632+0000","last_unstale":"2023-07-14T17:43:44.901014+0000","last_undegraded":"2023-07-14T17:43:44.901014+0000","last_fullsized":"2023-07-14T17:43:44.901014+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:23:03.050997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65561,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":443,"num_read_kb":292,"num_write":234,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":196683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":206,"ondisk_log_size":206,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481817,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":486248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.53900000000000003}]},{"osd":1,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":2.8330000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.39000000000000001}]},{"osd":2,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.36899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":4,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.44600000000000001}]},{"osd":5,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.499},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.52900000000000003}]},{"osd":6,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.65100000000000002}]},{"osd":7,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.621},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.51800000000000002}]},{"osd":8,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.63},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.64100000000000001}]},{"osd":9,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.65800000000000003}]},{"osd":10,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.68600000000000005}]}]},{"osd":10,"up_from":69,"seq":296352743452,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28652,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716500,"statfs":{"total":95995035648,"available":95965696000,"internally_reserved":0,"allocated":1671168,"data_stored":1122439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]},{"osd":1,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.47699999999999998}]},{"osd":2,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.92100000000000004}]},{"osd":3,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.72099999999999997}]},{"osd":4,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.51300000000000001}]},{"osd":5,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.82899999999999996}]},{"osd":6,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.75,"15min":0.75},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.86699999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":8,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.73599999999999999}]},{"osd":9,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.76600000000000001}]},{"osd":11,"last update":"Fri Jul 14 17:43:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.93200000000000005}]}]},{"osd":9,"up_from":64,"seq":274877906976,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":471246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.41099999999999998,"15min":0.42399999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.497,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.45400000000000001,"15min":0.47199999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.55800000000000005,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.39400000000000002}]},{"osd":1,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.59099999999999997,"15min":0.60599999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.69899999999999995,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.62},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.60199999999999998,"15min":0.61599999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70999999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.499}]},{"osd":2,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79900000000000004,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.64500000000000002,"15min":0.65700000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.73199999999999998,"5min":1.119,"15min":1.119},"last":0.64500000000000002}]},{"osd":3,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.48999999999999999,"15min":0.50600000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.52700000000000002,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.49199999999999999,"15min":0.505},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64400000000000002,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.34599999999999997}]},{"osd":4,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.628,"15min":0.64300000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.70699999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.63600000000000001,"15min":0.64800000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.68300000000000005,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.63400000000000001}]},{"osd":5,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.64600000000000002,"15min":0.65800000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.78400000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.65800000000000003,"15min":0.67100000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79600000000000004,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.66700000000000004}]},{"osd":6,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.372},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.52500000000000002}]},{"osd":7,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.70599999999999996,"15min":0.72799999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71099999999999997,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.71699999999999997,"15min":0.73799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.79400000000000004,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.57399999999999995}]},{"osd":8,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.749,"15min":0.76800000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.82699999999999996,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.74199999999999999,"15min":0.75800000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.85899999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.748}]},{"osd":10,"last update":"Fri Jul 14 17:43:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.73399999999999999,"15min":0.755},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.74399999999999999,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.73199999999999998,"15min":0.748},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.73999999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.65800000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]}]},{"osd":8,"up_from":57,"seq":244813135909,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":467528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.441},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53400000000000003,"15min":0.53300000000000003},"min":{"1min":0.35499999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.64100000000000001}]},{"osd":1,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.372,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83899999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.40999999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.39700000000000002}]},{"osd":2,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45800000000000002,"15min":0.46100000000000002},"min":{"1min":0.27500000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60599999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.47199999999999998,"15min":0.47999999999999998},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56499999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.54700000000000004}]},{"osd":3,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.36299999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.46000000000000002}]},{"osd":4,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85099999999999998,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64300000000000002,"15min":0.64200000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89700000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.53100000000000003}]},{"osd":5,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57399999999999995,"15min":0.58099999999999996},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72499999999999998,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.312,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.75,"5min":1.018,"15min":1.018},"last":0.33700000000000002}]},{"osd":6,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.66600000000000004,"15min":0.66800000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68500000000000005,"15min":0.69299999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84799999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.66400000000000003}]},{"osd":7,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.51000000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92400000000000004,"5min":1.008,"15min":1.008},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71099999999999997,"15min":0.70999999999999996},"min":{"1min":0.48899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88200000000000001,"5min":1.258,"15min":1.258},"last":0.74099999999999999}]},{"osd":9,"last update":"Fri Jul 14 17:43:19 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.68500000000000005,"15min":0.67000000000000004},"min":{"1min":0.441,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.68600000000000005,"15min":0.67200000000000004},"min":{"1min":0.46899999999999997,"5min":0.44,"15min":0.44},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.76000000000000001}]},{"osd":10,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.69699999999999995,"15min":0.68100000000000005},"min":{"1min":0.57199999999999995,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94399999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.70699999999999996,"15min":0.68500000000000005},"min":{"1min":0.61099999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.79700000000000004}]},{"osd":11,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.82499999999999996}]}]},{"osd":7,"up_from":49,"seq":210453397545,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1613824,"data_stored":1083346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.503},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.59499999999999997}]},{"osd":1,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.42199999999999999,"15min":0.42499999999999999},"min":{"1min":0.29999999999999999,"5min":0.25,"15min":0.25},"max":{"1min":0.70299999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.47299999999999998},"min":{"1min":0.32400000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69499999999999995,"5min":0.746,"15min":0.746},"last":0.56299999999999994}]},{"osd":2,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.64300000000000002,"15min":0.60799999999999998},"min":{"1min":0.443,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82799999999999996,"5min":5.1050000000000004,"15min":5.1050000000000004},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.59799999999999998,"15min":0.58599999999999997},"min":{"1min":0.47099999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.83699999999999997,"5min":1.048,"15min":1.048},"last":0.60999999999999999}]},{"osd":3,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.41799999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86499999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.60899999999999999},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80700000000000005,"5min":0.995,"15min":0.995},"last":0.58999999999999997}]},{"osd":4,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.505,"15min":0.503},"min":{"1min":0.36799999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.76300000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52000000000000002,"15min":0.53200000000000003},"min":{"1min":0.307,"5min":0.248,"15min":0.248},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.53700000000000003}]},{"osd":5,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.64800000000000002,"15min":0.63600000000000001},"min":{"1min":0.499,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98099999999999998,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.65600000000000003,"15min":0.64800000000000002},"min":{"1min":0.51600000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.96499999999999997,"5min":1.02,"15min":1.02},"last":0.63800000000000001}]},{"osd":6,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.64200000000000002,"15min":0.65000000000000002},"min":{"1min":0.45200000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.878,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.67600000000000005,"15min":0.68700000000000006},"min":{"1min":0.46700000000000003,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.94899999999999995,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.752}]},{"osd":8,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.45700000000000002,"5min":0.441,"15min":0.441},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.624},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.71299999999999997,"15min":0.72099999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0620000000000001,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.78300000000000003}]},{"osd":9,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.69399999999999995,"15min":0.70199999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.01,"5min":1.196,"15min":1.196},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.91200000000000003,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.76700000000000002}]},{"osd":10,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.65800000000000003,"15min":0.63700000000000001},"min":{"1min":0.53400000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.67700000000000005,"15min":0.66100000000000003},"min":{"1min":0.505,"5min":0.441,"15min":0.441},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.69899999999999995}]},{"osd":11,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.79700000000000004}]}]},{"osd":6,"up_from":44,"seq":188978561069,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":472116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.36799999999999999,"15min":0.35499999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.28199999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.307}]},{"osd":1,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57999999999999996,"15min":0.56799999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.76000000000000001,"5min":0.87,"15min":0.87},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.65000000000000002,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46000000000000002}]},{"osd":2,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59399999999999997,"15min":0.58999999999999997},"min":{"1min":0.44500000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.56399999999999995}]},{"osd":3,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.44400000000000001,"15min":0.44700000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68500000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.38},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.47199999999999998,"15min":0.45400000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78900000000000003,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.40400000000000003}]},{"osd":4,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85299999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.60799999999999998,"15min":0.60599999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81799999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.435}]},{"osd":5,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.60099999999999998,"15min":0.59299999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.68600000000000005,"15min":0.66300000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.8679999999999999,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.624}]},{"osd":7,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.754,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.66600000000000004,"15min":0.68000000000000005},"min":{"1min":0.47999999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.73499999999999999,"5min":1.012,"15min":1.012},"last":0.55500000000000005}]},{"osd":8,"last update":"Fri Jul 14 17:43:04 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.73599999999999999,"15min":0.755},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.98999999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.69499999999999995,"15min":0.71099999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92500000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.57399999999999995}]},{"osd":9,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.68100000000000005,"15min":0.70099999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.91500000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.65700000000000003,"15min":0.67700000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.90200000000000002,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.52300000000000002}]},{"osd":10,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.72799999999999998,"15min":0.746},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.81999999999999995,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.755,"15min":0.76900000000000002},"min":{"1min":0.52500000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.83799999999999997,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.64000000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":531697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.503,"15min":0.46700000000000003},"min":{"1min":0.33800000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.51500000000000001,"15min":0.47199999999999998},"min":{"1min":0.33600000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.78600000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Jul 14 17:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.54200000000000004,"15min":0.497},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.86099999999999999,"5min":0.879,"15min":0.879},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.52300000000000002,"15min":0.48899999999999999},"min":{"1min":0.36499999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.79200000000000004}]},{"osd":3,"last update":"Fri Jul 14 17:43:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53400000000000003,"15min":0.51600000000000001},"min":{"1min":0.31,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.75600000000000001,"5min":0.873,"15min":0.873},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57699999999999996,"15min":0.54500000000000004},"min":{"1min":0.44700000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.752,"5min":0.873,"15min":0.873},"last":0.63700000000000001}]},{"osd":4,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.45400000000000001,"15min":0.42999999999999999},"min":{"1min":0.34699999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.45700000000000002,"15min":0.433},"min":{"1min":0.29399999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.51900000000000002}]},{"osd":5,"last update":"Fri Jul 14 17:42:59 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.60899999999999999,"15min":0.58499999999999996},"min":{"1min":0.48299999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.58199999999999996,"15min":0.57099999999999995},"min":{"1min":0.42699999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.73399999999999999}]},{"osd":6,"last update":"Fri Jul 14 17:43:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.41599999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.66200000000000003}]},{"osd":7,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.56899999999999995,"15min":0.52800000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.55300000000000005,"15min":0.53300000000000003},"min":{"1min":0.373,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.70099999999999996}]},{"osd":8,"last update":"Fri Jul 14 17:42:59 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.65900000000000003,"15min":0.65600000000000003},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1,"5min":1,"15min":1},"last":0.75},{"interface":"front","average":{"1min":0.746,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.54000000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.88100000000000001}]},{"osd":9,"last update":"Fri Jul 14 17:43:19 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65500000000000003,"15min":0.65700000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.88400000000000001,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70199999999999996,"15min":0.70099999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.80800000000000005}]},{"osd":10,"last update":"Fri Jul 14 17:42:43 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.84299999999999997}]},{"osd":11,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.89500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":464062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 17:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.61499999999999999,"15min":0.53700000000000003},"min":{"1min":0.38500000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":8.5199999999999996,"5min":8.5199999999999996,"15min":8.5199999999999996},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.0960000000000001,"5min":0.65800000000000003,"15min":0.56000000000000005},"min":{"1min":0.41599999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":10.31,"5min":10.31,"15min":10.31},"last":0.46600000000000003}]},{"osd":2,"last update":"Fri Jul 14 17:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55900000000000005,"15min":0.53400000000000003},"min":{"1min":0.40999999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.502},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56999999999999995,"15min":0.55300000000000005},"min":{"1min":0.379,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.39200000000000002}]},{"osd":3,"last update":"Fri Jul 14 17:41:15 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44700000000000001,"15min":0.44600000000000001},"min":{"1min":0.28100000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.40999999999999998,"15min":0.40600000000000003},"min":{"1min":0.27900000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.61699999999999999,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.40799999999999997}]},{"osd":4,"last update":"Fri Jul 14 17:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55700000000000005,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":1.121,"5min":0.68100000000000005,"15min":0.58099999999999996},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":10.374000000000001,"5min":10.374000000000001,"15min":10.374000000000001},"last":0.54600000000000004}]},{"osd":5,"last update":"Fri Jul 14 17:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56499999999999995,"15min":0.55100000000000005},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.755,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59599999999999997,"15min":0.58999999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84099999999999997,"5min":0.88,"15min":0.88},"last":0.48899999999999999}]},{"osd":6,"last update":"Fri Jul 14 17:43:18 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.57999999999999996,"15min":0.502},"min":{"1min":0.30599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":10.468999999999999,"5min":10.468999999999999,"15min":10.468999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.58499999999999996,"15min":0.51600000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":8.4350000000000005,"5min":8.4350000000000005,"15min":8.4350000000000005},"last":0.377}]},{"osd":7,"last update":"Fri Jul 14 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.56399999999999995}]},{"osd":8,"last update":"Fri Jul 14 17:43:06 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.68500000000000005,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.82799999999999996,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.67100000000000004,"15min":0.67900000000000005},"min":{"1min":0.46200000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.89900000000000002,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.45200000000000001}]},{"osd":9,"last update":"Fri Jul 14 17:43:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.77600000000000002,"15min":0.72799999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":10.343,"5min":10.343,"15min":10.343},"last":0.42299999999999999},{"interface":"front","average":{"1min":1.5509999999999999,"5min":0.89000000000000001,"15min":0.78000000000000003},"min":{"1min":0.45300000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":18.739000000000001,"5min":18.739000000000001,"15min":18.739000000000001},"last":0.439}]},{"osd":10,"last update":"Fri Jul 14 17:42:44 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.57399999999999995}]},{"osd":11,"last update":"Fri Jul 14 17:43:06 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.53400000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":541552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.47499999999999998},"min":{"1min":0.41099999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.48299999999999998,"15min":0.46700000000000003},"min":{"1min":0.39100000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.59799999999999998}]},{"osd":1,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.53100000000000003,"15min":0.48299999999999998},"min":{"1min":0.35199999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.55300000000000005,"15min":0.50900000000000001},"min":{"1min":0.5,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.80100000000000005}]},{"osd":3,"last update":"Fri Jul 14 17:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.57499999999999996,"15min":0.54700000000000004},"min":{"1min":0.38100000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55900000000000005,"15min":0.53900000000000003},"min":{"1min":0.40100000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.86899999999999999,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.76900000000000002}]},{"osd":4,"last update":"Fri Jul 14 17:41:35 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.51900000000000002,"15min":0.51300000000000001},"min":{"1min":0.38400000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.55400000000000005,"15min":0.54600000000000004},"min":{"1min":0.39900000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.73799999999999999}]},{"osd":5,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.45600000000000002,"15min":0.43099999999999999},"min":{"1min":0.30399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.878},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.45200000000000001,"15min":0.437},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.89800000000000002}]},{"osd":6,"last update":"Fri Jul 14 17:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.60099999999999998,"15min":0.58099999999999996},"min":{"1min":0.47199999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.76000000000000001,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59899999999999998,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81799999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.85499999999999998}]},{"osd":7,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.59099999999999997,"15min":0.57499999999999996},"min":{"1min":0.40200000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.66000000000000003,"15min":0.64600000000000002},"min":{"1min":0.52500000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":1.0620000000000001}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.61299999999999999,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":1.006},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.59299999999999997,"15min":0.57399999999999995},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.96299999999999997}]},{"osd":9,"last update":"Fri Jul 14 17:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71399999999999997,"15min":0.70999999999999996},"min":{"1min":0.56999999999999995,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.995},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.76700000000000002,"15min":0.75600000000000001},"min":{"1min":0.64100000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":1.034}]},{"osd":10,"last update":"Fri Jul 14 17:42:41 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":1.046}]},{"osd":11,"last update":"Fri Jul 14 17:43:40 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":1.089},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":1.0209999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":481438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:16 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34300000000000003,"15min":0.32300000000000001},"min":{"1min":0.26900000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.437,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.36099999999999999,"15min":0.33200000000000002},"min":{"1min":0.28100000000000003,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.307}]},{"osd":1,"last update":"Fri Jul 14 17:43:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54200000000000004,"15min":0.50600000000000001},"min":{"1min":0.38600000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.82599999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54200000000000004,"15min":0.505},"min":{"1min":0.33300000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72499999999999998,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.34000000000000002}]},{"osd":2,"last update":"Fri Jul 14 17:43:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54700000000000004,"15min":0.52400000000000002},"min":{"1min":0.41899999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.69699999999999995,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.52800000000000002},"min":{"1min":0.40899999999999997,"5min":0.379,"15min":0.379},"max":{"1min":0.71999999999999997,"5min":1.423,"15min":1.423},"last":0.52800000000000002}]},{"osd":4,"last update":"Fri Jul 14 17:43:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58299999999999996,"15min":0.56699999999999995},"min":{"1min":0.378,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.748,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.44},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61199999999999999,"15min":0.58499999999999996},"min":{"1min":0.42499999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79400000000000004,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.57099999999999995}]},{"osd":5,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.38100000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85299999999999998,"5min":1.593,"15min":1.593},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.45400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.85699999999999998,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.59499999999999997}]},{"osd":6,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.52600000000000002,"15min":0.52900000000000003},"min":{"1min":0.32800000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.73699999999999999,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.46899999999999997,"15min":0.47599999999999998},"min":{"1min":0.32700000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.57399999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.54400000000000004}]},{"osd":7,"last update":"Fri Jul 14 17:43:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.61899999999999999,"15min":0.63700000000000001},"min":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82999999999999996,"5min":1.28,"15min":1.28},"last":0.505},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65500000000000003,"15min":0.64500000000000002},"min":{"1min":0.54400000000000004,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.82299999999999995,"5min":1.03,"15min":1.03},"last":0.63200000000000001}]},{"osd":8,"last update":"Fri Jul 14 17:41:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.55800000000000005}]},{"osd":9,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.878,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.68100000000000005,"15min":0.69699999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.60999999999999999}]},{"osd":10,"last update":"Fri Jul 14 17:43:44 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66200000000000003,"15min":0.67500000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.67700000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.749,"15min":0.76000000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.86799999999999999,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.66100000000000003}]},{"osd":11,"last update":"Fri Jul 14 17:43:44 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.67500000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953525,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":467070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.35399999999999998,"5min":0.311,"15min":0.311},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.52100000000000002,"15min":0.502},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.496}]},{"osd":1,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.41699999999999998,"15min":0.38200000000000001},"min":{"1min":0.372,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.61699999999999999,"5min":0.755,"15min":0.755},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.42699999999999999,"15min":0.40000000000000002},"min":{"1min":0.38,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.621,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.47799999999999998}]},{"osd":2,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.57499999999999996,"15min":0.56299999999999994},"min":{"1min":0.496,"5min":0.38,"15min":0.38},"max":{"1min":0.80200000000000005,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.622,"15min":0.58199999999999996},"min":{"1min":0.54400000000000004,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89600000000000002,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.86099999999999999}]},{"osd":3,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.59799999999999998,"15min":0.55100000000000005},"min":{"1min":0.44600000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.86599999999999999,"5min":1.498,"15min":1.498},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56200000000000006,"15min":0.53400000000000003},"min":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81100000000000005,"5min":1.629,"15min":1.629},"last":0.55000000000000004}]},{"osd":5,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.63100000000000001,"15min":0.61099999999999999},"min":{"1min":0.51500000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89800000000000002,"5min":1.256,"15min":1.256},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.624,"15min":0.60999999999999999},"min":{"1min":0.44900000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.68000000000000005}]},{"osd":6,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.72199999999999998,"15min":0.746},"min":{"1min":0.47699999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91800000000000004,"5min":2.7789999999999999,"15min":2.7789999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68200000000000005,"15min":0.66700000000000004},"min":{"1min":0.46600000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.95899999999999996,"5min":1.476,"15min":1.476},"last":0.71399999999999997}]},{"osd":7,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.57899999999999996,"15min":0.53300000000000003},"min":{"1min":0.49099999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.97299999999999998,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.61399999999999999,"15min":0.58399999999999996},"min":{"1min":0.5,"5min":0.377,"15min":0.377},"max":{"1min":0.88500000000000001,"5min":1.002,"15min":1.002},"last":0.73799999999999999}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.65600000000000003,"15min":0.64400000000000002},"min":{"1min":0.41999999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72299999999999998,"15min":0.71399999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.82699999999999996}]},{"osd":9,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.69399999999999995,"15min":0.69099999999999995},"min":{"1min":0.54000000000000004,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88100000000000001,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.80100000000000005,"15min":0.80400000000000005},"min":{"1min":0.61899999999999999,"5min":0.502,"15min":0.502},"max":{"1min":1.0049999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.78400000000000003}]},{"osd":10,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.82899999999999996,"15min":0.83099999999999996},"min":{"1min":0.63400000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0600000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.748,"5min":0.73199999999999998,"15min":0.72999999999999998},"min":{"1min":0.59799999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.75800000000000001}]},{"osd":11,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.93999999999999995}]}]},{"osd":5,"up_from":37,"seq":158913790002,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28588,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716564,"statfs":{"total":95995035648,"available":95965761536,"internally_reserved":0,"allocated":1605632,"data_stored":1069068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47099999999999997,"15min":0.44700000000000001},"min":{"1min":0.35499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77800000000000002,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.495},{"interface":"front","average":{"1min":0.501,"5min":0.48299999999999998,"15min":0.45700000000000002},"min":{"1min":0.35599999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65400000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.47499999999999998}]},{"osd":1,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.55100000000000005,"15min":0.497},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":5.3630000000000004,"5min":5.3630000000000004,"15min":5.3630000000000004},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51300000000000001,"15min":0.48999999999999999},"min":{"1min":0.42399999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.58599999999999997}]},{"osd":2,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.33500000000000002},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.379},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.376,"15min":0.35999999999999999},"min":{"1min":0.30499999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.40899999999999997}]},{"osd":3,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.501,"15min":0.495},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.51800000000000002,"15min":0.495},"min":{"1min":0.38700000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.747,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.60999999999999999}]},{"osd":4,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.53500000000000003,"15min":0.50800000000000001},"min":{"1min":0.434,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78100000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54300000000000004,"15min":0.52000000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.75600000000000001}]},{"osd":6,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.39000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.75700000000000001,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.627},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.56699999999999995,"15min":0.54300000000000004},"min":{"1min":0.46400000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80800000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.67900000000000005}]},{"osd":7,"last update":"Fri Jul 14 17:43:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.58899999999999997,"15min":0.57399999999999995},"min":{"1min":0.47599999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.63300000000000001,"15min":0.61299999999999999},"min":{"1min":0.51200000000000001,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.73899999999999999}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.54700000000000004,"15min":0.54000000000000004},"min":{"1min":0.38100000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.754,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.372,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.876,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.81299999999999994}]},{"osd":9,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.56999999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91000000000000003,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68400000000000005,"15min":0.67600000000000005},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.69399999999999995}]},{"osd":10,"last update":"Fri Jul 14 17:42:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":5.3330000000000002,"5min":5.3330000000000002,"15min":5.3330000000000002},"last":0.77700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":60629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T17:43:48.836 INFO:tasks.cephadm.ceph_manager.ceph:PG 4.6 is not active+clean 2023-07-14T17:43:48.836 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '4.6', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 88, 'state': 'activating', 'last_fresh': '2023-07-14T17:43:44.900723+0000', 'last_change': '2023-07-14T17:43:44.897633+0000', 'last_active': '2023-07-14T17:43:43.899017+0000', 'last_peered': '2023-07-14T17:42:50.079160+0000', 'last_clean': '2023-07-14T17:42:50.079160+0000', 'last_became_active': '2023-07-14T17:42:46.577067+0000', 'last_became_peered': '2023-07-14T17:42:46.577067+0000', 'last_unstale': '2023-07-14T17:43:44.900723+0000', 'last_undegraded': '2023-07-14T17:43:44.900723+0000', 'last_fullsized': '2023-07-14T17:43:44.900723+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 84, 'last_epoch_clean': 85, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-14T17:42:45.521346+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-14T17:42:45.521346+0000', 'last_clean_scrub_stamp': '2023-07-14T17:42:45.521346+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-07-15T22:06:30.535779+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 11, 6, 10], 'acting': [5, 11, 6, 10], 'avail_no_missing': ['5(0)', '6(2)', '9(1)', '10(3)', '11(1)'], 'object_location_counts': [], 'blocked_by': [2], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-07-14T17:43:48.837 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph status --format=json 2023-07-14T17:43:49.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:49 smithi178 ceph-mon[129368]: pgmap v290: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s; 0 B/s, 1 keys/s, 1 objects/s recovering 2023-07-14T17:43:49.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:43:49.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:43:49.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:49.417 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:49.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:49 smithi088 ceph-mon[127465]: pgmap v290: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s; 0 B/s, 1 keys/s, 1 objects/s recovering 2023-07-14T17:43:49.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:43:49.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:43:49.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:49.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:49 smithi107 ceph-mon[128999]: pgmap v290: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s; 0 B/s, 1 keys/s, 1 objects/s recovering 2023-07-14T17:43:49.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:43:49.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:43:49.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:43:50.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:50 smithi178 ceph-mon[129368]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:43:50.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:50 smithi088 ceph-mon[127465]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:43:50.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:50 smithi107 ceph-mon[128999]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:43:50.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.730+0000 7f5878a89700 1 -- 172.21.15.88:0/2216589304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587410a270 msgr2=0x7f587410c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:50.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.730+0000 7f5878a89700 1 --2- 172.21.15.88:0/2216589304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587410a270 0x7f587410c6b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f586c00b0a0 tx=0x7f586c00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:43:50.712 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.730+0000 7f5878a89700 1 -- 172.21.15.88:0/2216589304 shutdown_connections 2023-07-14T17:43:50.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.730+0000 7f5878a89700 1 --2- 172.21.15.88:0/2216589304 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587410a270 0x7f587410c6b0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:50.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.730+0000 7f5878a89700 1 --2- 172.21.15.88:0/2216589304 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5874100f20 0x7f58741012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:50.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.730+0000 7f5878a89700 1 --2- 172.21.15.88:0/2216589304 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5874101830 0x7f5874109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:50.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.730+0000 7f5878a89700 1 -- 172.21.15.88:0/2216589304 >> 172.21.15.88:0/2216589304 conn(0x7f5874075ca0 msgr2=0x7f58740760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:50.713 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.730+0000 7f5878a89700 1 -- 172.21.15.88:0/2216589304 shutdown_connections 2023-07-14T17:43:50.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.730+0000 7f5878a89700 1 -- 172.21.15.88:0/2216589304 wait complete. 2023-07-14T17:43:50.714 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.731+0000 7f5878a89700 1 Processor -- start 2023-07-14T17:43:50.716 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.731+0000 7f5878a89700 1 -- start start 2023-07-14T17:43:50.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.731+0000 7f5878a89700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5874100f20 0x7f5874195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:50.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.731+0000 7f5878a89700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5874101830 0x7f58741962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:50.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.731+0000 7f5878a89700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587410a270 0x7f587419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:50.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.731+0000 7f5878a89700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587419ad70 con 0x7f587410a270 2023-07-14T17:43:50.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.731+0000 7f5878a89700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587419aee0 con 0x7f5874101830 2023-07-14T17:43:50.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.731+0000 7f5878a89700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587419b0a0 con 0x7f5874100f20 2023-07-14T17:43:50.717 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.732+0000 7f5872d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587410a270 0x7f587419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:50.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.732+0000 7f5872d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587410a270 0x7f587419a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42468/0 (socket says 172.21.15.88:42468) 2023-07-14T17:43:50.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.732+0000 7f5872d9d700 1 -- 172.21.15.88:0/3579707567 learned_addr learned my addr 172.21.15.88:0/3579707567 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:50.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.732+0000 7f5871d9b700 1 --2- 172.21.15.88:0/3579707567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5874101830 0x7f58741962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:50.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.732+0000 7f587259c700 1 --2- 172.21.15.88:0/3579707567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5874100f20 0x7f5874195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:50.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.732+0000 7f5872d9d700 1 -- 172.21.15.88:0/3579707567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5874100f20 msgr2=0x7f5874195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:50.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.732+0000 7f5872d9d700 1 --2- 172.21.15.88:0/3579707567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5874100f20 0x7f5874195db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:50.718 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.732+0000 7f5872d9d700 1 -- 172.21.15.88:0/3579707567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5874101830 msgr2=0x7f58741962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:50.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.732+0000 7f5872d9d700 1 --2- 172.21.15.88:0/3579707567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5874101830 0x7f58741962f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:50.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.732+0000 7f5872d9d700 1 -- 172.21.15.88:0/3579707567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586c009d00 con 0x7f587410a270 2023-07-14T17:43:50.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.732+0000 7f587259c700 1 --2- 172.21.15.88:0/3579707567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5874100f20 0x7f5874195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:43:50.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.732+0000 7f5872d9d700 1 --2- 172.21.15.88:0/3579707567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587410a270 0x7f587419a760 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f586c00b830 tx=0x7f586c0094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:50.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.732+0000 7f58637fe700 1 -- 172.21.15.88:0/3579707567 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586c00e040 con 0x7f587410a270 2023-07-14T17:43:50.719 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.732+0000 7f5878a89700 1 -- 172.21.15.88:0/3579707567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f587419b320 con 0x7f587410a270 2023-07-14T17:43:50.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.732+0000 7f58637fe700 1 -- 172.21.15.88:0/3579707567 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f586c003e90 con 0x7f587410a270 2023-07-14T17:43:50.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.732+0000 7f58637fe700 1 -- 172.21.15.88:0/3579707567 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f586c004880 con 0x7f587410a270 2023-07-14T17:43:50.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.732+0000 7f5878a89700 1 -- 172.21.15.88:0/3579707567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58741a2be0 con 0x7f587410a270 2023-07-14T17:43:50.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.733+0000 7f5878a89700 1 -- 172.21.15.88:0/3579707567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5874104b60 con 0x7f587410a270 2023-07-14T17:43:50.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.734+0000 7f58637fe700 1 -- 172.21.15.88:0/3579707567 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f586c004000 con 0x7f587410a270 2023-07-14T17:43:50.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.735+0000 7f58637fe700 1 --2- 172.21.15.88:0/3579707567 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f585c0a1260 0x7f585c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:50.720 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.735+0000 7f587259c700 1 --2- 172.21.15.88:0/3579707567 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f585c0a1260 0x7f585c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:50.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.735+0000 7f58637fe700 1 -- 172.21.15.88:0/3579707567 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f586c0cf040 con 0x7f587410a270 2023-07-14T17:43:50.721 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.735+0000 7f587259c700 1 --2- 172.21.15.88:0/3579707567 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f585c0a1260 0x7f585c0a3710 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5868005fd0 tx=0x7f5868005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:50.722 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:50.738+0000 7f58637fe700 1 -- 172.21.15.88:0/3579707567 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f586c098350 con 0x7f587410a270 2023-07-14T17:43:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:51.001+0000 7f5878a89700 1 -- 172.21.15.88:0/3579707567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f587404e340 con 0x7f587410a270 2023-07-14T17:43:50.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:51.001+0000 7f58637fe700 1 -- 172.21.15.88:0/3579707567 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+3218 (secure 0 0 0) 0x7f586c09b970 con 0x7f587410a270 2023-07-14T17:43:50.984 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:43:50.984 INFO:teuthology.orchestra.run.smithi088.stdout:{"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":375,"monmap":{"epoch":3,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":88,"num_osds":12,"num_up_osds":12,"osd_up_since":1689356521,"num_in_osds":12,"osd_in_since":1689356503,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":128},{"state_name":"activating","count":1}],"num_pgs":129,"num_pools":4,"num_objects":102,"data_bytes":656175,"bytes_used":346120192,"bytes_avail":1151594307584,"bytes_total":1151940427776,"inactive_pgs_ratio":0.0077519379556179047,"recovering_objects_per_sec":1,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":1,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":10,"read_bytes_sec":3839,"read_op_per_sec":7,"write_op_per_sec":0},"fsmap":{"epoch":27,"id":1,"up":5,"in":5,"max":5,"by_rank":[{"filesystem_id":1,"rank":4,"name":"g","status":"up:standby-replay","gid":14616},{"filesystem_id":1,"rank":3,"name":"a","status":"up:standby-replay","gid":14628},{"filesystem_id":1,"rank":2,"name":"l","status":"up:standby-replay","gid":24415},{"filesystem_id":1,"rank":1,"name":"k","status":"up:standby-replay","gid":24419},{"filesystem_id":1,"rank":0,"name":"f","status":"up:standby-replay","gid":24442},{"filesystem_id":1,"rank":4,"name":"e","status":"up:active","gid":24449},{"filesystem_id":1,"rank":3,"name":"c","status":"up:active","gid":24454},{"filesystem_id":1,"rank":2,"name":"i","status":"up:active","gid":24469},{"filesystem_id":1,"rank":1,"name":"h","status":"up:active","gid":24473},{"filesystem_id":1,"rank":0,"name":"b","status":"up:active","gid":24506}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":2,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.88:8443/"}},"servicemap":{"epoch":21,"modified":"2023-07-14T17:43:49.888413+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"e":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"f":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"g":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"h":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"i":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"k":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"l":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-07-14T17:43:50.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:51.003+0000 7f5878a89700 1 -- 172.21.15.88:0/3579707567 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f585c0a1260 msgr2=0x7f585c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:50.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:51.003+0000 7f5878a89700 1 --2- 172.21.15.88:0/3579707567 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f585c0a1260 0x7f585c0a3710 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5868005fd0 tx=0x7f5868005dc0 comp rx=0 tx=0).stop 2023-07-14T17:43:50.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:51.003+0000 7f5878a89700 1 -- 172.21.15.88:0/3579707567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587410a270 msgr2=0x7f587419a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:50.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:51.003+0000 7f5878a89700 1 --2- 172.21.15.88:0/3579707567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587410a270 0x7f587419a760 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f586c00b830 tx=0x7f586c0094c0 comp rx=0 tx=0).stop 2023-07-14T17:43:50.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:51.004+0000 7f5878a89700 1 -- 172.21.15.88:0/3579707567 shutdown_connections 2023-07-14T17:43:50.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:51.004+0000 7f5878a89700 1 --2- 172.21.15.88:0/3579707567 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f585c0a1260 0x7f585c0a3710 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:50.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:51.004+0000 7f5878a89700 1 --2- 172.21.15.88:0/3579707567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f587410a270 0x7f587419a760 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:50.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:51.004+0000 7f5878a89700 1 --2- 172.21.15.88:0/3579707567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5874100f20 0x7f5874195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:50.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:51.004+0000 7f5878a89700 1 --2- 172.21.15.88:0/3579707567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5874101830 0x7f58741962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:50.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:51.004+0000 7f5878a89700 1 -- 172.21.15.88:0/3579707567 >> 172.21.15.88:0/3579707567 conn(0x7f5874075ca0 msgr2=0x7f5874101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:50.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:51.004+0000 7f5878a89700 1 -- 172.21.15.88:0/3579707567 shutdown_connections 2023-07-14T17:43:50.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:51.004+0000 7f5878a89700 1 -- 172.21.15.88:0/3579707567 wait complete. 2023-07-14T17:43:51.328 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:51 smithi088 ceph-mon[127465]: pgmap v291: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 1 keys/s, 1 objects/s recovering 2023-07-14T17:43:51.329 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:51 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3579707567' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2023-07-14T17:43:51.375 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2023-07-14T17:43:51.376 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T17:43:51.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:51 smithi178 ceph-mon[129368]: pgmap v291: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 1 keys/s, 1 objects/s recovering 2023-07-14T17:43:51.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:51 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3579707567' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2023-07-14T17:43:51.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:51 smithi107 ceph-mon[128999]: pgmap v291: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 0 B/s, 1 keys/s, 1 objects/s recovering 2023-07-14T17:43:51.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:51 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3579707567' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2023-07-14T17:43:51.927 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:53.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.158+0000 7f51ca4a8700 1 -- 172.21.15.88:0/2619003268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51c41007a0 msgr2=0x7f51c4100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:53.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.158+0000 7f51ca4a8700 1 --2- 172.21.15.88:0/2619003268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51c41007a0 0x7f51c4100c10 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f51b80099c0 tx=0x7f51b8009cd0 comp rx=0 tx=0).stop 2023-07-14T17:43:53.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.159+0000 7f51ca4a8700 1 -- 172.21.15.88:0/2619003268 shutdown_connections 2023-07-14T17:43:53.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.159+0000 7f51ca4a8700 1 --2- 172.21.15.88:0/2619003268 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51c4101150 0x7f51c410bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:53.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.159+0000 7f51ca4a8700 1 --2- 172.21.15.88:0/2619003268 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51c40fef40 0x7f51c40ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:53.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.159+0000 7f51ca4a8700 1 --2- 172.21.15.88:0/2619003268 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51c41007a0 0x7f51c4100c10 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:53.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.159+0000 7f51ca4a8700 1 -- 172.21.15.88:0/2619003268 >> 172.21.15.88:0/2619003268 conn(0x7f51c4075c90 msgr2=0x7f51c4076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:53.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.159+0000 7f51ca4a8700 1 -- 172.21.15.88:0/2619003268 shutdown_connections 2023-07-14T17:43:53.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.159+0000 7f51ca4a8700 1 -- 172.21.15.88:0/2619003268 wait complete. 2023-07-14T17:43:53.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.159+0000 7f51ca4a8700 1 Processor -- start 2023-07-14T17:43:53.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.159+0000 7f51ca4a8700 1 -- start start 2023-07-14T17:43:53.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51ca4a8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51c40fef40 0x7f51c4101980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:53.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51ca4a8700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51c41007a0 0x7f51c4101ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:53.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51ca4a8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51c4101150 0x7f51c4102440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:53.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51ca4a8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c41a41b0 con 0x7f51c4101150 2023-07-14T17:43:53.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51ca4a8700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c41a4320 con 0x7f51c40fef40 2023-07-14T17:43:53.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51ca4a8700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c41a44e0 con 0x7f51c41007a0 2023-07-14T17:43:53.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51c8a45700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51c4101150 0x7f51c4102440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:53.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51c8a45700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51c4101150 0x7f51c4102440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42480/0 (socket says 172.21.15.88:42480) 2023-07-14T17:43:53.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51c8a45700 1 -- 172.21.15.88:0/1220866897 learned_addr learned my addr 172.21.15.88:0/1220866897 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:53.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51c3fff700 1 --2- 172.21.15.88:0/1220866897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51c40fef40 0x7f51c4101980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:53.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51c37fe700 1 --2- 172.21.15.88:0/1220866897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51c41007a0 0x7f51c4101ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:53.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51c8a45700 1 -- 172.21.15.88:0/1220866897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51c41007a0 msgr2=0x7f51c4101ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:53.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51c8a45700 1 --2- 172.21.15.88:0/1220866897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51c41007a0 0x7f51c4101ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:53.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51c8a45700 1 -- 172.21.15.88:0/1220866897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51c40fef40 msgr2=0x7f51c4101980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:53.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51c8a45700 1 --2- 172.21.15.88:0/1220866897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51c40fef40 0x7f51c4101980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:53.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51c8a45700 1 -- 172.21.15.88:0/1220866897 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b80095f0 con 0x7f51c4101150 2023-07-14T17:43:53.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51c8a45700 1 --2- 172.21.15.88:0/1220866897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51c4101150 0x7f51c4102440 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00ada0 tx=0x7f51bc00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:53.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51c17fa700 1 -- 172.21.15.88:0/1220866897 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51bc00f070 con 0x7f51c4101150 2023-07-14T17:43:53.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51c17fa700 1 -- 172.21.15.88:0/1220866897 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f51bc007b30 con 0x7f51c4101150 2023-07-14T17:43:53.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.160+0000 7f51ca4a8700 1 -- 172.21.15.88:0/1220866897 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c41a47c0 con 0x7f51c4101150 2023-07-14T17:43:53.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.161+0000 7f51ca4a8700 1 -- 172.21.15.88:0/1220866897 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c41a4c40 con 0x7f51c4101150 2023-07-14T17:43:53.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.161+0000 7f51c17fa700 1 -- 172.21.15.88:0/1220866897 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51bc005280 con 0x7f51c4101150 2023-07-14T17:43:53.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.162+0000 7f51ca4a8700 1 -- 172.21.15.88:0/1220866897 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c404e340 con 0x7f51c4101150 2023-07-14T17:43:53.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.163+0000 7f51c17fa700 1 -- 172.21.15.88:0/1220866897 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f51bc00c8d0 con 0x7f51c4101150 2023-07-14T17:43:53.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.163+0000 7f51c17fa700 1 --2- 172.21.15.88:0/1220866897 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f51ac0a11e0 0x7f51ac0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:53.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.163+0000 7f51c3fff700 1 --2- 172.21.15.88:0/1220866897 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f51ac0a11e0 0x7f51ac0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:53.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.163+0000 7f51c17fa700 1 -- 172.21.15.88:0/1220866897 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f51bc0cc320 con 0x7f51c4101150 2023-07-14T17:43:53.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.164+0000 7f51c3fff700 1 --2- 172.21.15.88:0/1220866897 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f51ac0a11e0 0x7f51ac0a3690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f51b4005950 tx=0x7f51b400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:53.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.167+0000 7f51c17fa700 1 -- 172.21.15.88:0/1220866897 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f51bc095690 con 0x7f51c4101150 2023-07-14T17:43:53.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:53 smithi088 ceph-mon[127465]: pgmap v292: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s; 0 B/s, 1 keys/s, 1 objects/s recovering 2023-07-14T17:43:53.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.349+0000 7f51ca4a8700 1 -- 172.21.15.88:0/1220866897 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f51c4069140 con 0x7f51ac0a11e0 2023-07-14T17:43:53.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.361+0000 7f51c17fa700 1 -- 172.21.15.88:0/1220866897 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427079 (secure 0 0 0) 0x7f51c4069140 con 0x7f51ac0a11e0 2023-07-14T17:43:53.347 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:43:53.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.365+0000 7f51ca4a8700 1 -- 172.21.15.88:0/1220866897 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f51ac0a11e0 msgr2=0x7f51ac0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:53.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.365+0000 7f51ca4a8700 1 --2- 172.21.15.88:0/1220866897 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f51ac0a11e0 0x7f51ac0a3690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f51b4005950 tx=0x7f51b400b410 comp rx=0 tx=0).stop 2023-07-14T17:43:53.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.365+0000 7f51ca4a8700 1 -- 172.21.15.88:0/1220866897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51c4101150 msgr2=0x7f51c4102440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:53.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.365+0000 7f51ca4a8700 1 --2- 172.21.15.88:0/1220866897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51c4101150 0x7f51c4102440 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00ada0 tx=0x7f51bc00c330 comp rx=0 tx=0).stop 2023-07-14T17:43:53.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.365+0000 7f51ca4a8700 1 -- 172.21.15.88:0/1220866897 shutdown_connections 2023-07-14T17:43:53.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.365+0000 7f51ca4a8700 1 --2- 172.21.15.88:0/1220866897 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f51ac0a11e0 0x7f51ac0a3690 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:53.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.365+0000 7f51ca4a8700 1 --2- 172.21.15.88:0/1220866897 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f51c4101150 0x7f51c4102440 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:53.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.365+0000 7f51ca4a8700 1 --2- 172.21.15.88:0/1220866897 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f51c40fef40 0x7f51c4101980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:53.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.365+0000 7f51ca4a8700 1 --2- 172.21.15.88:0/1220866897 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f51c41007a0 0x7f51c4101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:53.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.365+0000 7f51ca4a8700 1 -- 172.21.15.88:0/1220866897 >> 172.21.15.88:0/1220866897 conn(0x7f51c4075c90 msgr2=0x7f51c4068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:53.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.365+0000 7f51ca4a8700 1 -- 172.21.15.88:0/1220866897 shutdown_connections 2023-07-14T17:43:53.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:53.365+0000 7f51ca4a8700 1 -- 172.21.15.88:0/1220866897 wait complete. 2023-07-14T17:43:53.351 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-07-14T17:43:53.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:53 smithi178 ceph-mon[129368]: pgmap v292: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s; 0 B/s, 1 keys/s, 1 objects/s recovering 2023-07-14T17:43:53.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:53 smithi107 ceph-mon[128999]: pgmap v292: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s; 0 B/s, 1 keys/s, 1 objects/s recovering 2023-07-14T17:43:53.675 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2023-07-14T17:43:51.888786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656175,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":619,"num_read_kb":426,"num_write":417,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":345,"ondisk_log_size":345,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":420,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":338068,"kb_used_data":13792,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124603756,"statfs":{"total":1151940427776,"available":1151594246144,"internally_reserved":0,"allocated":14123008,"data_stored":7772378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":90,"num_read_kb":45,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001977"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892404+0000","last_change":"2023-07-14T17:42:42.540396+0000","last_active":"2023-07-14T17:43:44.892404+0000","last_peered":"2023-07-14T17:43:44.892404+0000","last_clean":"2023-07-14T17:43:44.892404+0000","last_became_active":"2023-07-14T17:42:42.539732+0000","last_became_peered":"2023-07-14T17:42:42.539732+0000","last_unstale":"2023-07-14T17:43:44.892404+0000","last_undegraded":"2023-07-14T17:43:44.892404+0000","last_fullsized":"2023-07-14T17:43:44.892404+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:45:35.673679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.899803+0000","last_change":"2023-07-14T17:42:42.532153+0000","last_active":"2023-07-14T17:43:44.899803+0000","last_peered":"2023-07-14T17:43:44.899803+0000","last_clean":"2023-07-14T17:43:44.899803+0000","last_became_active":"2023-07-14T17:42:42.531745+0000","last_became_peered":"2023-07-14T17:42:42.531745+0000","last_unstale":"2023-07-14T17:43:44.899803+0000","last_undegraded":"2023-07-14T17:43:44.899803+0000","last_fullsized":"2023-07-14T17:43:44.899803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:38:17.037985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.898523+0000","last_change":"2023-07-14T17:42:42.549497+0000","last_active":"2023-07-14T17:43:44.898523+0000","last_peered":"2023-07-14T17:43:44.898523+0000","last_clean":"2023-07-14T17:43:44.898523+0000","last_became_active":"2023-07-14T17:42:42.549201+0000","last_became_peered":"2023-07-14T17:42:42.549201+0000","last_unstale":"2023-07-14T17:43:44.898523+0000","last_undegraded":"2023-07-14T17:43:44.898523+0000","last_fullsized":"2023-07-14T17:43:44.898523+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:07:07.859324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.814335+0000","last_change":"2023-07-14T17:42:42.543460+0000","last_active":"2023-07-14T17:43:45.814335+0000","last_peered":"2023-07-14T17:43:45.814335+0000","last_clean":"2023-07-14T17:43:45.814335+0000","last_became_active":"2023-07-14T17:42:42.543127+0000","last_became_peered":"2023-07-14T17:42:42.543127+0000","last_unstale":"2023-07-14T17:43:45.814335+0000","last_undegraded":"2023-07-14T17:43:45.814335+0000","last_fullsized":"2023-07-14T17:43:45.814335+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:06:06.644157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894474+0000","last_change":"2023-07-14T17:42:42.534242+0000","last_active":"2023-07-14T17:43:44.894474+0000","last_peered":"2023-07-14T17:43:44.894474+0000","last_clean":"2023-07-14T17:43:44.894474+0000","last_became_active":"2023-07-14T17:42:42.533536+0000","last_became_peered":"2023-07-14T17:42:42.533536+0000","last_unstale":"2023-07-14T17:43:44.894474+0000","last_undegraded":"2023-07-14T17:43:44.894474+0000","last_fullsized":"2023-07-14T17:43:44.894474+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:53:30.057794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896541+0000","last_change":"2023-07-14T17:42:42.538042+0000","last_active":"2023-07-14T17:43:44.896541+0000","last_peered":"2023-07-14T17:43:44.896541+0000","last_clean":"2023-07-14T17:43:44.896541+0000","last_became_active":"2023-07-14T17:42:42.537612+0000","last_became_peered":"2023-07-14T17:42:42.537612+0000","last_unstale":"2023-07-14T17:43:44.896541+0000","last_undegraded":"2023-07-14T17:43:44.896541+0000","last_fullsized":"2023-07-14T17:43:44.896541+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:29:37.412187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894322+0000","last_change":"2023-07-14T17:42:42.545318+0000","last_active":"2023-07-14T17:43:44.894322+0000","last_peered":"2023-07-14T17:43:44.894322+0000","last_clean":"2023-07-14T17:43:44.894322+0000","last_became_active":"2023-07-14T17:42:42.544984+0000","last_became_peered":"2023-07-14T17:42:42.544984+0000","last_unstale":"2023-07-14T17:43:44.894322+0000","last_undegraded":"2023-07-14T17:43:44.894322+0000","last_fullsized":"2023-07-14T17:43:44.894322+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:10:59.223910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.900595+0000","last_change":"2023-07-14T17:43:44.899856+0000","last_active":"2023-07-14T17:43:44.900595+0000","last_peered":"2023-07-14T17:43:44.900595+0000","last_clean":"2023-07-14T17:43:44.900595+0000","last_became_active":"2023-07-14T17:43:44.899478+0000","last_became_peered":"2023-07-14T17:43:44.899478+0000","last_unstale":"2023-07-14T17:43:44.900595+0000","last_undegraded":"2023-07-14T17:43:44.900595+0000","last_fullsized":"2023-07-14T17:43:44.900595+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:32:02.556140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897879+0000","last_change":"2023-07-14T17:42:42.548973+0000","last_active":"2023-07-14T17:43:44.897879+0000","last_peered":"2023-07-14T17:43:44.897879+0000","last_clean":"2023-07-14T17:43:44.897879+0000","last_became_active":"2023-07-14T17:42:42.548672+0000","last_became_peered":"2023-07-14T17:42:42.548672+0000","last_unstale":"2023-07-14T17:43:44.897879+0000","last_undegraded":"2023-07-14T17:43:44.897879+0000","last_fullsized":"2023-07-14T17:43:44.897879+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:55:26.825661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896875+0000","last_change":"2023-07-14T17:42:42.540969+0000","last_active":"2023-07-14T17:43:44.896875+0000","last_peered":"2023-07-14T17:43:44.896875+0000","last_clean":"2023-07-14T17:43:44.896875+0000","last_became_active":"2023-07-14T17:42:42.540653+0000","last_became_peered":"2023-07-14T17:42:42.540653+0000","last_unstale":"2023-07-14T17:43:44.896875+0000","last_undegraded":"2023-07-14T17:43:44.896875+0000","last_fullsized":"2023-07-14T17:43:44.896875+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:07.062764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894491+0000","last_change":"2023-07-14T17:42:42.550016+0000","last_active":"2023-07-14T17:43:44.894491+0000","last_peered":"2023-07-14T17:43:44.894491+0000","last_clean":"2023-07-14T17:43:44.894491+0000","last_became_active":"2023-07-14T17:42:42.549653+0000","last_became_peered":"2023-07-14T17:42:42.549653+0000","last_unstale":"2023-07-14T17:43:44.894491+0000","last_undegraded":"2023-07-14T17:43:44.894491+0000","last_fullsized":"2023-07-14T17:43:44.894491+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:43.271416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.814147+0000","last_change":"2023-07-14T17:42:42.552105+0000","last_active":"2023-07-14T17:43:45.814147+0000","last_peered":"2023-07-14T17:43:45.814147+0000","last_clean":"2023-07-14T17:43:45.814147+0000","last_became_active":"2023-07-14T17:42:42.550830+0000","last_became_peered":"2023-07-14T17:42:42.550830+0000","last_unstale":"2023-07-14T17:43:45.814147+0000","last_undegraded":"2023-07-14T17:43:45.814147+0000","last_fullsized":"2023-07-14T17:43:45.814147+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:08:48.231339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324361+0000","last_change":"2023-07-14T17:42:42.534963+0000","last_active":"2023-07-14T17:43:45.324361+0000","last_peered":"2023-07-14T17:43:45.324361+0000","last_clean":"2023-07-14T17:43:45.324361+0000","last_became_active":"2023-07-14T17:42:42.534546+0000","last_became_peered":"2023-07-14T17:42:42.534546+0000","last_unstale":"2023-07-14T17:43:45.324361+0000","last_undegraded":"2023-07-14T17:43:45.324361+0000","last_fullsized":"2023-07-14T17:43:45.324361+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:37:07.860214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.898319+0000","last_change":"2023-07-14T17:42:42.530073+0000","last_active":"2023-07-14T17:43:44.898319+0000","last_peered":"2023-07-14T17:43:44.898319+0000","last_clean":"2023-07-14T17:43:44.898319+0000","last_became_active":"2023-07-14T17:42:42.529767+0000","last_became_peered":"2023-07-14T17:42:42.529767+0000","last_unstale":"2023-07-14T17:43:44.898319+0000","last_undegraded":"2023-07-14T17:43:44.898319+0000","last_fullsized":"2023-07-14T17:43:44.898319+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:07:57.028886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.898063+0000","last_change":"2023-07-14T17:42:42.548168+0000","last_active":"2023-07-14T17:43:44.898063+0000","last_peered":"2023-07-14T17:43:44.898063+0000","last_clean":"2023-07-14T17:43:44.898063+0000","last_became_active":"2023-07-14T17:42:42.547363+0000","last_became_peered":"2023-07-14T17:42:42.547363+0000","last_unstale":"2023-07-14T17:43:44.898063+0000","last_undegraded":"2023-07-14T17:43:44.898063+0000","last_fullsized":"2023-07-14T17:43:44.898063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:43:45.670787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895946+0000","last_change":"2023-07-14T17:42:42.551042+0000","last_active":"2023-07-14T17:43:44.895946+0000","last_peered":"2023-07-14T17:43:44.895946+0000","last_clean":"2023-07-14T17:43:44.895946+0000","last_became_active":"2023-07-14T17:42:42.550730+0000","last_became_peered":"2023-07-14T17:42:42.550730+0000","last_unstale":"2023-07-14T17:43:44.895946+0000","last_undegraded":"2023-07-14T17:43:44.895946+0000","last_fullsized":"2023-07-14T17:43:44.895946+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:33:43.890169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892180+0000","last_change":"2023-07-14T17:42:42.540528+0000","last_active":"2023-07-14T17:43:44.892180+0000","last_peered":"2023-07-14T17:43:44.892180+0000","last_clean":"2023-07-14T17:43:44.892180+0000","last_became_active":"2023-07-14T17:42:42.540093+0000","last_became_peered":"2023-07-14T17:42:42.540093+0000","last_unstale":"2023-07-14T17:43:44.892180+0000","last_undegraded":"2023-07-14T17:43:44.892180+0000","last_fullsized":"2023-07-14T17:43:44.892180+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:13:34.468585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.893607+0000","last_change":"2023-07-14T17:42:42.539512+0000","last_active":"2023-07-14T17:43:44.893607+0000","last_peered":"2023-07-14T17:43:44.893607+0000","last_clean":"2023-07-14T17:43:44.893607+0000","last_became_active":"2023-07-14T17:42:42.539248+0000","last_became_peered":"2023-07-14T17:42:42.539248+0000","last_unstale":"2023-07-14T17:43:44.893607+0000","last_undegraded":"2023-07-14T17:43:44.893607+0000","last_fullsized":"2023-07-14T17:43:44.893607+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:32:15.064855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895813+0000","last_change":"2023-07-14T17:42:42.531630+0000","last_active":"2023-07-14T17:43:44.895813+0000","last_peered":"2023-07-14T17:43:44.895813+0000","last_clean":"2023-07-14T17:43:44.895813+0000","last_became_active":"2023-07-14T17:42:42.531333+0000","last_became_peered":"2023-07-14T17:42:42.531333+0000","last_unstale":"2023-07-14T17:43:44.895813+0000","last_undegraded":"2023-07-14T17:43:44.895813+0000","last_fullsized":"2023-07-14T17:43:44.895813+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:37:35.706668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894477+0000","last_change":"2023-07-14T17:42:46.543062+0000","last_active":"2023-07-14T17:43:44.894477+0000","last_peered":"2023-07-14T17:43:44.894477+0000","last_clean":"2023-07-14T17:43:44.894477+0000","last_became_active":"2023-07-14T17:42:46.542189+0000","last_became_peered":"2023-07-14T17:42:46.542189+0000","last_unstale":"2023-07-14T17:43:44.894477+0000","last_undegraded":"2023-07-14T17:43:44.894477+0000","last_fullsized":"2023-07-14T17:43:44.894477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:57:03.881795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":84,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:46.859961+0000","last_change":"2023-07-14T17:42:41.914589+0000","last_active":"2023-07-14T17:43:46.859961+0000","last_peered":"2023-07-14T17:43:46.859961+0000","last_clean":"2023-07-14T17:43:46.859961+0000","last_became_active":"2023-07-14T17:42:41.914275+0000","last_became_peered":"2023-07-14T17:42:41.914275+0000","last_unstale":"2023-07-14T17:43:46.859961+0000","last_undegraded":"2023-07-14T17:43:46.859961+0000","last_fullsized":"2023-07-14T17:43:46.859961+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:16.066793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":45,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.900792+0000","last_change":"2023-07-14T17:43:44.900792+0000","last_active":"2023-07-14T17:43:44.900792+0000","last_peered":"2023-07-14T17:43:44.900792+0000","last_clean":"2023-07-14T17:43:44.900792+0000","last_became_active":"2023-07-14T17:43:44.900333+0000","last_became_peered":"2023-07-14T17:43:44.900333+0000","last_unstale":"2023-07-14T17:43:44.900792+0000","last_undegraded":"2023-07-14T17:43:44.900792+0000","last_fullsized":"2023-07-14T17:43:44.900792+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:29:20.451913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.813589+0000","last_change":"2023-07-14T17:42:46.542500+0000","last_active":"2023-07-14T17:43:45.813589+0000","last_peered":"2023-07-14T17:43:45.813589+0000","last_clean":"2023-07-14T17:43:45.813589+0000","last_became_active":"2023-07-14T17:42:46.541932+0000","last_became_peered":"2023-07-14T17:42:46.541932+0000","last_unstale":"2023-07-14T17:43:45.813589+0000","last_undegraded":"2023-07-14T17:43:45.813589+0000","last_fullsized":"2023-07-14T17:43:45.813589+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:49:41.767435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897058+0000","last_change":"2023-07-14T17:42:41.918169+0000","last_active":"2023-07-14T17:43:44.897058+0000","last_peered":"2023-07-14T17:43:44.897058+0000","last_clean":"2023-07-14T17:43:44.897058+0000","last_became_active":"2023-07-14T17:42:41.917124+0000","last_became_peered":"2023-07-14T17:42:41.917124+0000","last_unstale":"2023-07-14T17:43:44.897058+0000","last_undegraded":"2023-07-14T17:43:44.897058+0000","last_fullsized":"2023-07-14T17:43:44.897058+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:05:43.117229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.898688+0000","last_change":"2023-07-14T17:42:42.539265+0000","last_active":"2023-07-14T17:43:44.898688+0000","last_peered":"2023-07-14T17:43:44.898688+0000","last_clean":"2023-07-14T17:43:44.898688+0000","last_became_active":"2023-07-14T17:42:42.538933+0000","last_became_peered":"2023-07-14T17:42:42.538933+0000","last_unstale":"2023-07-14T17:43:44.898688+0000","last_undegraded":"2023-07-14T17:43:44.898688+0000","last_fullsized":"2023-07-14T17:43:44.898688+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:30:34.884083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.813943+0000","last_change":"2023-07-14T17:42:46.550194+0000","last_active":"2023-07-14T17:43:45.813943+0000","last_peered":"2023-07-14T17:43:45.813943+0000","last_clean":"2023-07-14T17:43:45.813943+0000","last_became_active":"2023-07-14T17:42:46.549693+0000","last_became_peered":"2023-07-14T17:42:46.549693+0000","last_unstale":"2023-07-14T17:43:45.813943+0000","last_undegraded":"2023-07-14T17:43:45.813943+0000","last_fullsized":"2023-07-14T17:43:45.813943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:03:33.891030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.908529+0000","last_change":"2023-07-14T17:43:44.908253+0000","last_active":"2023-07-14T17:43:44.908529+0000","last_peered":"2023-07-14T17:43:44.908529+0000","last_clean":"2023-07-14T17:43:44.908529+0000","last_became_active":"2023-07-14T17:43:44.896490+0000","last_became_peered":"2023-07-14T17:43:44.896490+0000","last_unstale":"2023-07-14T17:43:44.908529+0000","last_undegraded":"2023-07-14T17:43:44.908529+0000","last_fullsized":"2023-07-14T17:43:44.908529+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:21:55.326452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896660+0000","last_change":"2023-07-14T17:42:42.543846+0000","last_active":"2023-07-14T17:43:44.896660+0000","last_peered":"2023-07-14T17:43:44.896660+0000","last_clean":"2023-07-14T17:43:44.896660+0000","last_became_active":"2023-07-14T17:42:42.543528+0000","last_became_peered":"2023-07-14T17:42:42.543528+0000","last_unstale":"2023-07-14T17:43:44.896660+0000","last_undegraded":"2023-07-14T17:43:44.896660+0000","last_fullsized":"2023-07-14T17:43:44.896660+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:14:43.217768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.813751+0000","last_change":"2023-07-14T17:42:46.547799+0000","last_active":"2023-07-14T17:43:45.813751+0000","last_peered":"2023-07-14T17:43:45.813751+0000","last_clean":"2023-07-14T17:43:45.813751+0000","last_became_active":"2023-07-14T17:42:46.547381+0000","last_became_peered":"2023-07-14T17:42:46.547381+0000","last_unstale":"2023-07-14T17:43:45.813751+0000","last_undegraded":"2023-07-14T17:43:45.813751+0000","last_fullsized":"2023-07-14T17:43:45.813751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:06:35.931658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895253+0000","last_change":"2023-07-14T17:42:41.914911+0000","last_active":"2023-07-14T17:43:44.895253+0000","last_peered":"2023-07-14T17:43:44.895253+0000","last_clean":"2023-07-14T17:43:44.895253+0000","last_became_active":"2023-07-14T17:42:41.914607+0000","last_became_peered":"2023-07-14T17:42:41.914607+0000","last_unstale":"2023-07-14T17:43:44.895253+0000","last_undegraded":"2023-07-14T17:43:44.895253+0000","last_fullsized":"2023-07-14T17:43:44.895253+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:34:12.929058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897636+0000","last_change":"2023-07-14T17:42:42.548866+0000","last_active":"2023-07-14T17:43:44.897636+0000","last_peered":"2023-07-14T17:43:44.897636+0000","last_clean":"2023-07-14T17:43:44.897636+0000","last_became_active":"2023-07-14T17:42:42.548562+0000","last_became_peered":"2023-07-14T17:42:42.548562+0000","last_unstale":"2023-07-14T17:43:44.897636+0000","last_undegraded":"2023-07-14T17:43:44.897636+0000","last_fullsized":"2023-07-14T17:43:44.897636+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:41:33.093669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895655+0000","last_change":"2023-07-14T17:42:46.538879+0000","last_active":"2023-07-14T17:43:44.895655+0000","last_peered":"2023-07-14T17:43:44.895655+0000","last_clean":"2023-07-14T17:43:44.895655+0000","last_became_active":"2023-07-14T17:42:46.538476+0000","last_became_peered":"2023-07-14T17:42:46.538476+0000","last_unstale":"2023-07-14T17:43:44.895655+0000","last_undegraded":"2023-07-14T17:43:44.895655+0000","last_fullsized":"2023-07-14T17:43:44.895655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:50:18.995823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":80,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:48.849702+0000","last_change":"2023-07-14T17:42:41.917215+0000","last_active":"2023-07-14T17:43:48.849702+0000","last_peered":"2023-07-14T17:43:48.849702+0000","last_clean":"2023-07-14T17:43:48.849702+0000","last_became_active":"2023-07-14T17:42:41.916640+0000","last_became_peered":"2023-07-14T17:42:41.916640+0000","last_unstale":"2023-07-14T17:43:48.849702+0000","last_undegraded":"2023-07-14T17:43:48.849702+0000","last_fullsized":"2023-07-14T17:43:48.849702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:57:44.899889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897271+0000","last_change":"2023-07-14T17:42:42.534347+0000","last_active":"2023-07-14T17:43:44.897271+0000","last_peered":"2023-07-14T17:43:44.897271+0000","last_clean":"2023-07-14T17:43:44.897271+0000","last_became_active":"2023-07-14T17:42:42.533958+0000","last_became_peered":"2023-07-14T17:42:42.533958+0000","last_unstale":"2023-07-14T17:43:44.897271+0000","last_undegraded":"2023-07-14T17:43:44.897271+0000","last_fullsized":"2023-07-14T17:43:44.897271+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:35:41.285730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894270+0000","last_change":"2023-07-14T17:42:46.543144+0000","last_active":"2023-07-14T17:43:44.894270+0000","last_peered":"2023-07-14T17:43:44.894270+0000","last_clean":"2023-07-14T17:43:44.894270+0000","last_became_active":"2023-07-14T17:42:46.542311+0000","last_became_peered":"2023-07-14T17:42:46.542311+0000","last_unstale":"2023-07-14T17:43:44.894270+0000","last_undegraded":"2023-07-14T17:43:44.894270+0000","last_fullsized":"2023-07-14T17:43:44.894270+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:21:37.953172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896529+0000","last_change":"2023-07-14T17:42:41.916174+0000","last_active":"2023-07-14T17:43:44.896529+0000","last_peered":"2023-07-14T17:43:44.896529+0000","last_clean":"2023-07-14T17:43:44.896529+0000","last_became_active":"2023-07-14T17:42:41.915867+0000","last_became_peered":"2023-07-14T17:42:41.915867+0000","last_unstale":"2023-07-14T17:43:44.896529+0000","last_undegraded":"2023-07-14T17:43:44.896529+0000","last_fullsized":"2023-07-14T17:43:44.896529+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:59:36.014596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897789+0000","last_change":"2023-07-14T17:42:42.547064+0000","last_active":"2023-07-14T17:43:44.897789+0000","last_peered":"2023-07-14T17:43:44.897789+0000","last_clean":"2023-07-14T17:43:44.897789+0000","last_became_active":"2023-07-14T17:42:42.546777+0000","last_became_peered":"2023-07-14T17:42:42.546777+0000","last_unstale":"2023-07-14T17:43:44.897789+0000","last_undegraded":"2023-07-14T17:43:44.897789+0000","last_fullsized":"2023-07-14T17:43:44.897789+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:44:45.781755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892451+0000","last_change":"2023-07-14T17:42:46.548647+0000","last_active":"2023-07-14T17:43:44.892451+0000","last_peered":"2023-07-14T17:43:44.892451+0000","last_clean":"2023-07-14T17:43:44.892451+0000","last_became_active":"2023-07-14T17:42:46.548142+0000","last_became_peered":"2023-07-14T17:42:46.548142+0000","last_unstale":"2023-07-14T17:43:44.892451+0000","last_undegraded":"2023-07-14T17:43:44.892451+0000","last_fullsized":"2023-07-14T17:43:44.892451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:50:46.460689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324145+0000","last_change":"2023-07-14T17:42:41.911675+0000","last_active":"2023-07-14T17:43:45.324145+0000","last_peered":"2023-07-14T17:43:45.324145+0000","last_clean":"2023-07-14T17:43:45.324145+0000","last_became_active":"2023-07-14T17:42:41.911299+0000","last_became_peered":"2023-07-14T17:42:41.911299+0000","last_unstale":"2023-07-14T17:43:45.324145+0000","last_undegraded":"2023-07-14T17:43:45.324145+0000","last_fullsized":"2023-07-14T17:43:45.324145+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:37:15.112003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895160+0000","last_change":"2023-07-14T17:42:42.536211+0000","last_active":"2023-07-14T17:43:44.895160+0000","last_peered":"2023-07-14T17:43:44.895160+0000","last_clean":"2023-07-14T17:43:44.895160+0000","last_became_active":"2023-07-14T17:42:42.535763+0000","last_became_peered":"2023-07-14T17:42:42.535763+0000","last_unstale":"2023-07-14T17:43:44.895160+0000","last_undegraded":"2023-07-14T17:43:44.895160+0000","last_fullsized":"2023-07-14T17:43:44.895160+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:14:23.840550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.899792+0000","last_change":"2023-07-14T17:42:46.540917+0000","last_active":"2023-07-14T17:43:44.899792+0000","last_peered":"2023-07-14T17:43:44.899792+0000","last_clean":"2023-07-14T17:43:44.899792+0000","last_became_active":"2023-07-14T17:42:46.540575+0000","last_became_peered":"2023-07-14T17:42:46.540575+0000","last_unstale":"2023-07-14T17:43:44.899792+0000","last_undegraded":"2023-07-14T17:43:44.899792+0000","last_fullsized":"2023-07-14T17:43:44.899792+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:01:47.539703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897945+0000","last_change":"2023-07-14T17:42:41.917646+0000","last_active":"2023-07-14T17:43:44.897945+0000","last_peered":"2023-07-14T17:43:44.897945+0000","last_clean":"2023-07-14T17:43:44.897945+0000","last_became_active":"2023-07-14T17:42:41.917331+0000","last_became_peered":"2023-07-14T17:42:41.917331+0000","last_unstale":"2023-07-14T17:43:44.897945+0000","last_undegraded":"2023-07-14T17:43:44.897945+0000","last_fullsized":"2023-07-14T17:43:44.897945+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:52:16.441394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892768+0000","last_change":"2023-07-14T17:42:42.526696+0000","last_active":"2023-07-14T17:43:44.892768+0000","last_peered":"2023-07-14T17:43:44.892768+0000","last_clean":"2023-07-14T17:43:44.892768+0000","last_became_active":"2023-07-14T17:42:42.526353+0000","last_became_peered":"2023-07-14T17:42:42.526353+0000","last_unstale":"2023-07-14T17:43:44.892768+0000","last_undegraded":"2023-07-14T17:43:44.892768+0000","last_fullsized":"2023-07-14T17:43:44.892768+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:07:24.895596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895396+0000","last_change":"2023-07-14T17:42:46.545460+0000","last_active":"2023-07-14T17:43:44.895396+0000","last_peered":"2023-07-14T17:43:44.895396+0000","last_clean":"2023-07-14T17:43:44.895396+0000","last_became_active":"2023-07-14T17:42:46.544897+0000","last_became_peered":"2023-07-14T17:42:46.544897+0000","last_unstale":"2023-07-14T17:43:44.895396+0000","last_undegraded":"2023-07-14T17:43:44.895396+0000","last_fullsized":"2023-07-14T17:43:44.895396+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:59:41.696162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":122,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:49.850566+0000","last_change":"2023-07-14T17:42:41.912266+0000","last_active":"2023-07-14T17:43:49.850566+0000","last_peered":"2023-07-14T17:43:49.850566+0000","last_clean":"2023-07-14T17:43:49.850566+0000","last_became_active":"2023-07-14T17:42:41.911944+0000","last_became_peered":"2023-07-14T17:42:41.911944+0000","last_unstale":"2023-07-14T17:43:49.850566+0000","last_undegraded":"2023-07-14T17:43:49.850566+0000","last_fullsized":"2023-07-14T17:43:49.850566+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:17:02.788197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":92,"num_read_kb":92,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897607+0000","last_change":"2023-07-14T17:42:42.548609+0000","last_active":"2023-07-14T17:43:44.897607+0000","last_peered":"2023-07-14T17:43:44.897607+0000","last_clean":"2023-07-14T17:43:44.897607+0000","last_became_active":"2023-07-14T17:42:42.548253+0000","last_became_peered":"2023-07-14T17:42:42.548253+0000","last_unstale":"2023-07-14T17:43:44.897607+0000","last_undegraded":"2023-07-14T17:43:44.897607+0000","last_fullsized":"2023-07-14T17:43:44.897607+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:21:18.974575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892477+0000","last_change":"2023-07-14T17:42:46.547324+0000","last_active":"2023-07-14T17:43:44.892477+0000","last_peered":"2023-07-14T17:43:44.892477+0000","last_clean":"2023-07-14T17:43:44.892477+0000","last_became_active":"2023-07-14T17:42:46.546876+0000","last_became_peered":"2023-07-14T17:42:46.546876+0000","last_unstale":"2023-07-14T17:43:44.892477+0000","last_undegraded":"2023-07-14T17:43:44.892477+0000","last_fullsized":"2023-07-14T17:43:44.892477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:07:48.996897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.818146+0000","last_change":"2023-07-14T17:43:45.818146+0000","last_active":"2023-07-14T17:43:45.818146+0000","last_peered":"2023-07-14T17:43:45.818146+0000","last_clean":"2023-07-14T17:43:45.818146+0000","last_became_active":"2023-07-14T17:43:45.814954+0000","last_became_peered":"2023-07-14T17:43:45.814954+0000","last_unstale":"2023-07-14T17:43:45.818146+0000","last_undegraded":"2023-07-14T17:43:45.818146+0000","last_fullsized":"2023-07-14T17:43:45.818146+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:50:05.262289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.898727+0000","last_change":"2023-07-14T17:42:42.548081+0000","last_active":"2023-07-14T17:43:44.898727+0000","last_peered":"2023-07-14T17:43:44.898727+0000","last_clean":"2023-07-14T17:43:44.898727+0000","last_became_active":"2023-07-14T17:42:42.547876+0000","last_became_peered":"2023-07-14T17:42:42.547876+0000","last_unstale":"2023-07-14T17:43:44.898727+0000","last_undegraded":"2023-07-14T17:43:44.898727+0000","last_fullsized":"2023-07-14T17:43:44.898727+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:11:34.698640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.814077+0000","last_change":"2023-07-14T17:42:46.550516+0000","last_active":"2023-07-14T17:43:45.814077+0000","last_peered":"2023-07-14T17:43:45.814077+0000","last_clean":"2023-07-14T17:43:45.814077+0000","last_became_active":"2023-07-14T17:42:46.549588+0000","last_became_peered":"2023-07-14T17:42:46.549588+0000","last_unstale":"2023-07-14T17:43:45.814077+0000","last_undegraded":"2023-07-14T17:43:45.814077+0000","last_fullsized":"2023-07-14T17:43:45.814077+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:55:54.065571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897608+0000","last_change":"2023-07-14T17:42:41.917527+0000","last_active":"2023-07-14T17:43:44.897608+0000","last_peered":"2023-07-14T17:43:44.897608+0000","last_clean":"2023-07-14T17:43:44.897608+0000","last_became_active":"2023-07-14T17:42:41.917239+0000","last_became_peered":"2023-07-14T17:42:41.917239+0000","last_unstale":"2023-07-14T17:43:44.897608+0000","last_undegraded":"2023-07-14T17:43:44.897608+0000","last_fullsized":"2023-07-14T17:43:44.897608+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:07:11.390441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896841+0000","last_change":"2023-07-14T17:42:42.554866+0000","last_active":"2023-07-14T17:43:44.896841+0000","last_peered":"2023-07-14T17:43:44.896841+0000","last_clean":"2023-07-14T17:43:44.896841+0000","last_became_active":"2023-07-14T17:42:42.554585+0000","last_became_peered":"2023-07-14T17:42:42.554585+0000","last_unstale":"2023-07-14T17:43:44.896841+0000","last_undegraded":"2023-07-14T17:43:44.896841+0000","last_fullsized":"2023-07-14T17:43:44.896841+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:38:59.633305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892595+0000","last_change":"2023-07-14T17:42:46.544788+0000","last_active":"2023-07-14T17:43:44.892595+0000","last_peered":"2023-07-14T17:43:44.892595+0000","last_clean":"2023-07-14T17:43:44.892595+0000","last_became_active":"2023-07-14T17:42:46.544419+0000","last_became_peered":"2023-07-14T17:42:46.544419+0000","last_unstale":"2023-07-14T17:43:44.892595+0000","last_undegraded":"2023-07-14T17:43:44.892595+0000","last_fullsized":"2023-07-14T17:43:44.892595+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:26:19.151208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"86'33","reported_seq":157,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:49.850569+0000","last_change":"2023-07-14T17:42:41.917069+0000","last_active":"2023-07-14T17:43:49.850569+0000","last_peered":"2023-07-14T17:43:49.850569+0000","last_clean":"2023-07-14T17:43:49.850569+0000","last_became_active":"2023-07-14T17:42:41.916630+0000","last_became_peered":"2023-07-14T17:42:41.916630+0000","last_unstale":"2023-07-14T17:43:49.850569+0000","last_undegraded":"2023-07-14T17:43:49.850569+0000","last_fullsized":"2023-07-14T17:43:49.850569+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:52:18.568627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58551,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":98,"num_read_kb":105,"num_write":35,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896482+0000","last_change":"2023-07-14T17:42:42.542124+0000","last_active":"2023-07-14T17:43:44.896482+0000","last_peered":"2023-07-14T17:43:44.896482+0000","last_clean":"2023-07-14T17:43:44.896482+0000","last_became_active":"2023-07-14T17:42:42.541851+0000","last_became_peered":"2023-07-14T17:42:42.541851+0000","last_unstale":"2023-07-14T17:43:44.896482+0000","last_undegraded":"2023-07-14T17:43:44.896482+0000","last_fullsized":"2023-07-14T17:43:44.896482+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:56:52.567761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897932+0000","last_change":"2023-07-14T17:42:46.546167+0000","last_active":"2023-07-14T17:43:44.897932+0000","last_peered":"2023-07-14T17:43:44.897932+0000","last_clean":"2023-07-14T17:43:44.897932+0000","last_became_active":"2023-07-14T17:42:46.545768+0000","last_became_peered":"2023-07-14T17:42:46.545768+0000","last_unstale":"2023-07-14T17:43:44.897932+0000","last_undegraded":"2023-07-14T17:43:44.897932+0000","last_fullsized":"2023-07-14T17:43:44.897932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:13:37.105161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894811+0000","last_change":"2023-07-14T17:42:41.918858+0000","last_active":"2023-07-14T17:43:44.894811+0000","last_peered":"2023-07-14T17:43:44.894811+0000","last_clean":"2023-07-14T17:43:44.894811+0000","last_became_active":"2023-07-14T17:42:41.918568+0000","last_became_peered":"2023-07-14T17:42:41.918568+0000","last_unstale":"2023-07-14T17:43:44.894811+0000","last_undegraded":"2023-07-14T17:43:44.894811+0000","last_fullsized":"2023-07-14T17:43:44.894811+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:43:09.338997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324334+0000","last_change":"2023-07-14T17:42:42.542310+0000","last_active":"2023-07-14T17:43:45.324334+0000","last_peered":"2023-07-14T17:43:45.324334+0000","last_clean":"2023-07-14T17:43:45.324334+0000","last_became_active":"2023-07-14T17:42:42.541989+0000","last_became_peered":"2023-07-14T17:42:42.541989+0000","last_unstale":"2023-07-14T17:43:45.324334+0000","last_undegraded":"2023-07-14T17:43:45.324334+0000","last_fullsized":"2023-07-14T17:43:45.324334+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:23:26.638546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.325836+0000","last_change":"2023-07-14T17:43:45.325836+0000","last_active":"2023-07-14T17:43:45.325836+0000","last_peered":"2023-07-14T17:43:45.325836+0000","last_clean":"2023-07-14T17:43:45.325836+0000","last_became_active":"2023-07-14T17:43:45.325330+0000","last_became_peered":"2023-07-14T17:43:45.325330+0000","last_unstale":"2023-07-14T17:43:45.325836+0000","last_undegraded":"2023-07-14T17:43:45.325836+0000","last_fullsized":"2023-07-14T17:43:45.325836+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:34:58.970503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897638+0000","last_change":"2023-07-14T17:42:41.916382+0000","last_active":"2023-07-14T17:43:44.897638+0000","last_peered":"2023-07-14T17:43:44.897638+0000","last_clean":"2023-07-14T17:43:44.897638+0000","last_became_active":"2023-07-14T17:42:41.916082+0000","last_became_peered":"2023-07-14T17:42:41.916082+0000","last_unstale":"2023-07-14T17:43:44.897638+0000","last_undegraded":"2023-07-14T17:43:44.897638+0000","last_fullsized":"2023-07-14T17:43:44.897638+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:28:56.714038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894470+0000","last_change":"2023-07-14T17:42:42.546812+0000","last_active":"2023-07-14T17:43:44.894470+0000","last_peered":"2023-07-14T17:43:44.894470+0000","last_clean":"2023-07-14T17:43:44.894470+0000","last_became_active":"2023-07-14T17:42:42.546517+0000","last_became_peered":"2023-07-14T17:42:42.546517+0000","last_unstale":"2023-07-14T17:43:44.894470+0000","last_undegraded":"2023-07-14T17:43:44.894470+0000","last_fullsized":"2023-07-14T17:43:44.894470+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:01:05.671027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.893117+0000","last_change":"2023-07-14T17:42:46.547726+0000","last_active":"2023-07-14T17:43:44.893117+0000","last_peered":"2023-07-14T17:43:44.893117+0000","last_clean":"2023-07-14T17:43:44.893117+0000","last_became_active":"2023-07-14T17:42:46.547207+0000","last_became_peered":"2023-07-14T17:42:46.547207+0000","last_unstale":"2023-07-14T17:43:44.893117+0000","last_undegraded":"2023-07-14T17:43:44.893117+0000","last_fullsized":"2023-07-14T17:43:44.893117+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:17:09.830803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"86'6","reported_seq":78,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:46.859084+0000","last_change":"2023-07-14T17:42:41.913802+0000","last_active":"2023-07-14T17:43:46.859084+0000","last_peered":"2023-07-14T17:43:46.859084+0000","last_clean":"2023-07-14T17:43:46.859084+0000","last_became_active":"2023-07-14T17:42:41.913467+0000","last_became_peered":"2023-07-14T17:42:41.913467+0000","last_unstale":"2023-07-14T17:43:46.859084+0000","last_undegraded":"2023-07-14T17:43:46.859084+0000","last_fullsized":"2023-07-14T17:43:46.859084+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:28:23.910132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":47,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894173+0000","last_change":"2023-07-14T17:42:42.545379+0000","last_active":"2023-07-14T17:43:44.894173+0000","last_peered":"2023-07-14T17:43:44.894173+0000","last_clean":"2023-07-14T17:43:44.894173+0000","last_became_active":"2023-07-14T17:42:42.545012+0000","last_became_peered":"2023-07-14T17:42:42.545012+0000","last_unstale":"2023-07-14T17:43:44.894173+0000","last_undegraded":"2023-07-14T17:43:44.894173+0000","last_fullsized":"2023-07-14T17:43:44.894173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:53:26.889969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324438+0000","last_change":"2023-07-14T17:42:46.544524+0000","last_active":"2023-07-14T17:43:45.324438+0000","last_peered":"2023-07-14T17:43:45.324438+0000","last_clean":"2023-07-14T17:43:45.324438+0000","last_became_active":"2023-07-14T17:42:46.544152+0000","last_became_peered":"2023-07-14T17:42:46.544152+0000","last_unstale":"2023-07-14T17:43:45.324438+0000","last_undegraded":"2023-07-14T17:43:45.324438+0000","last_fullsized":"2023-07-14T17:43:45.324438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:15:31.705985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":44,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894178+0000","last_change":"2023-07-14T17:42:41.917282+0000","last_active":"2023-07-14T17:43:44.894178+0000","last_peered":"2023-07-14T17:43:44.894178+0000","last_clean":"2023-07-14T17:43:44.894178+0000","last_became_active":"2023-07-14T17:42:41.916994+0000","last_became_peered":"2023-07-14T17:42:41.916994+0000","last_unstale":"2023-07-14T17:43:44.894178+0000","last_undegraded":"2023-07-14T17:43:44.894178+0000","last_fullsized":"2023-07-14T17:43:44.894178+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:44:02.404111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897640+0000","last_change":"2023-07-14T17:42:42.548525+0000","last_active":"2023-07-14T17:43:44.897640+0000","last_peered":"2023-07-14T17:43:44.897640+0000","last_clean":"2023-07-14T17:43:44.897640+0000","last_became_active":"2023-07-14T17:42:42.548270+0000","last_became_peered":"2023-07-14T17:42:42.548270+0000","last_unstale":"2023-07-14T17:43:44.897640+0000","last_undegraded":"2023-07-14T17:43:44.897640+0000","last_fullsized":"2023-07-14T17:43:44.897640+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:14:31.331118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.898128+0000","last_change":"2023-07-14T17:42:46.546480+0000","last_active":"2023-07-14T17:43:44.898128+0000","last_peered":"2023-07-14T17:43:44.898128+0000","last_clean":"2023-07-14T17:43:44.898128+0000","last_became_active":"2023-07-14T17:42:46.546078+0000","last_became_peered":"2023-07-14T17:42:46.546078+0000","last_unstale":"2023-07-14T17:43:44.898128+0000","last_undegraded":"2023-07-14T17:43:44.898128+0000","last_fullsized":"2023-07-14T17:43:44.898128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:41:01.115708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894258+0000","last_change":"2023-07-14T17:42:41.914226+0000","last_active":"2023-07-14T17:43:44.894258+0000","last_peered":"2023-07-14T17:43:44.894258+0000","last_clean":"2023-07-14T17:43:44.894258+0000","last_became_active":"2023-07-14T17:42:41.913820+0000","last_became_peered":"2023-07-14T17:42:41.913820+0000","last_unstale":"2023-07-14T17:43:44.894258+0000","last_undegraded":"2023-07-14T17:43:44.894258+0000","last_fullsized":"2023-07-14T17:43:44.894258+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:21:15.638447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892019+0000","last_change":"2023-07-14T17:42:42.543549+0000","last_active":"2023-07-14T17:43:44.892019+0000","last_peered":"2023-07-14T17:43:44.892019+0000","last_clean":"2023-07-14T17:43:44.892019+0000","last_became_active":"2023-07-14T17:42:42.543245+0000","last_became_peered":"2023-07-14T17:42:42.543245+0000","last_unstale":"2023-07-14T17:43:44.892019+0000","last_undegraded":"2023-07-14T17:43:44.892019+0000","last_fullsized":"2023-07-14T17:43:44.892019+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:18:13.987338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897150+0000","last_change":"2023-07-14T17:42:46.577245+0000","last_active":"2023-07-14T17:43:44.897150+0000","last_peered":"2023-07-14T17:43:44.897150+0000","last_clean":"2023-07-14T17:43:44.897150+0000","last_became_active":"2023-07-14T17:42:46.576793+0000","last_became_peered":"2023-07-14T17:42:46.576793+0000","last_unstale":"2023-07-14T17:43:44.897150+0000","last_undegraded":"2023-07-14T17:43:44.897150+0000","last_fullsized":"2023-07-14T17:43:44.897150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:22:16.800489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892975+0000","last_change":"2023-07-14T17:42:41.917199+0000","last_active":"2023-07-14T17:43:44.892975+0000","last_peered":"2023-07-14T17:43:44.892975+0000","last_clean":"2023-07-14T17:43:44.892975+0000","last_became_active":"2023-07-14T17:42:41.916924+0000","last_became_peered":"2023-07-14T17:42:41.916924+0000","last_unstale":"2023-07-14T17:43:44.892975+0000","last_undegraded":"2023-07-14T17:43:44.892975+0000","last_fullsized":"2023-07-14T17:43:44.892975+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:10:08.589818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896869+0000","last_change":"2023-07-14T17:42:42.540531+0000","last_active":"2023-07-14T17:43:44.896869+0000","last_peered":"2023-07-14T17:43:44.896869+0000","last_clean":"2023-07-14T17:43:44.896869+0000","last_became_active":"2023-07-14T17:42:42.540165+0000","last_became_peered":"2023-07-14T17:42:42.540165+0000","last_unstale":"2023-07-14T17:43:44.896869+0000","last_undegraded":"2023-07-14T17:43:44.896869+0000","last_fullsized":"2023-07-14T17:43:44.896869+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:30:36.053357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324213+0000","last_change":"2023-07-14T17:42:46.537158+0000","last_active":"2023-07-14T17:43:45.324213+0000","last_peered":"2023-07-14T17:43:45.324213+0000","last_clean":"2023-07-14T17:43:45.324213+0000","last_became_active":"2023-07-14T17:42:46.536816+0000","last_became_peered":"2023-07-14T17:42:46.536816+0000","last_unstale":"2023-07-14T17:43:45.324213+0000","last_undegraded":"2023-07-14T17:43:45.324213+0000","last_fullsized":"2023-07-14T17:43:45.324213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:00:28.047931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":79,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:49.863661+0000","last_change":"2023-07-14T17:42:41.913638+0000","last_active":"2023-07-14T17:43:49.863661+0000","last_peered":"2023-07-14T17:43:49.863661+0000","last_clean":"2023-07-14T17:43:49.863661+0000","last_became_active":"2023-07-14T17:42:41.913361+0000","last_became_peered":"2023-07-14T17:42:41.913361+0000","last_unstale":"2023-07-14T17:43:49.863661+0000","last_undegraded":"2023-07-14T17:43:49.863661+0000","last_fullsized":"2023-07-14T17:43:49.863661+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:46:04.106529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.901414+0000","last_change":"2023-07-14T17:43:44.901414+0000","last_active":"2023-07-14T17:43:44.901414+0000","last_peered":"2023-07-14T17:43:44.901414+0000","last_clean":"2023-07-14T17:43:44.901414+0000","last_became_active":"2023-07-14T17:43:44.901164+0000","last_became_peered":"2023-07-14T17:43:44.901164+0000","last_unstale":"2023-07-14T17:43:44.901414+0000","last_undegraded":"2023-07-14T17:43:44.901414+0000","last_fullsized":"2023-07-14T17:43:44.901414+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:15:25.526757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324220+0000","last_change":"2023-07-14T17:42:46.576999+0000","last_active":"2023-07-14T17:43:45.324220+0000","last_peered":"2023-07-14T17:43:45.324220+0000","last_clean":"2023-07-14T17:43:45.324220+0000","last_became_active":"2023-07-14T17:42:46.576652+0000","last_became_peered":"2023-07-14T17:42:46.576652+0000","last_unstale":"2023-07-14T17:43:45.324220+0000","last_undegraded":"2023-07-14T17:43:45.324220+0000","last_fullsized":"2023-07-14T17:43:45.324220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:14.124797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897667+0000","last_change":"2023-07-14T17:42:42.547053+0000","last_active":"2023-07-14T17:43:44.897667+0000","last_peered":"2023-07-14T17:43:44.897667+0000","last_clean":"2023-07-14T17:43:44.897667+0000","last_became_active":"2023-07-14T17:42:42.546735+0000","last_became_peered":"2023-07-14T17:42:42.546735+0000","last_unstale":"2023-07-14T17:43:44.897667+0000","last_undegraded":"2023-07-14T17:43:44.897667+0000","last_fullsized":"2023-07-14T17:43:44.897667+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:48:04.951463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":78,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:48.861787+0000","last_change":"2023-07-14T17:42:41.918692+0000","last_active":"2023-07-14T17:43:48.861787+0000","last_peered":"2023-07-14T17:43:48.861787+0000","last_clean":"2023-07-14T17:43:48.861787+0000","last_became_active":"2023-07-14T17:42:41.918360+0000","last_became_peered":"2023-07-14T17:42:41.918360+0000","last_unstale":"2023-07-14T17:43:48.861787+0000","last_undegraded":"2023-07-14T17:43:48.861787+0000","last_fullsized":"2023-07-14T17:43:48.861787+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:13:06.540310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":47,"num_read_kb":47,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894327+0000","last_change":"2023-07-14T17:42:46.543997+0000","last_active":"2023-07-14T17:43:44.894327+0000","last_peered":"2023-07-14T17:43:44.894327+0000","last_clean":"2023-07-14T17:43:44.894327+0000","last_became_active":"2023-07-14T17:42:46.543650+0000","last_became_peered":"2023-07-14T17:42:46.543650+0000","last_unstale":"2023-07-14T17:43:44.894327+0000","last_undegraded":"2023-07-14T17:43:44.894327+0000","last_fullsized":"2023-07-14T17:43:44.894327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:30:06.964324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895872+0000","last_change":"2023-07-14T17:42:42.548860+0000","last_active":"2023-07-14T17:43:44.895872+0000","last_peered":"2023-07-14T17:43:44.895872+0000","last_clean":"2023-07-14T17:43:44.895872+0000","last_became_active":"2023-07-14T17:42:42.548564+0000","last_became_peered":"2023-07-14T17:42:42.548564+0000","last_unstale":"2023-07-14T17:43:44.895872+0000","last_undegraded":"2023-07-14T17:43:44.895872+0000","last_fullsized":"2023-07-14T17:43:44.895872+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:51:19.885558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.813974+0000","last_change":"2023-07-14T17:42:41.912601+0000","last_active":"2023-07-14T17:43:45.813974+0000","last_peered":"2023-07-14T17:43:45.813974+0000","last_clean":"2023-07-14T17:43:45.813974+0000","last_became_active":"2023-07-14T17:42:41.912265+0000","last_became_peered":"2023-07-14T17:42:41.912265+0000","last_unstale":"2023-07-14T17:43:45.813974+0000","last_undegraded":"2023-07-14T17:43:45.813974+0000","last_fullsized":"2023-07-14T17:43:45.813974+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:20:08.059396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":347,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894007+0000","last_change":"2023-07-14T17:41:43.696093+0000","last_active":"2023-07-14T17:43:44.894007+0000","last_peered":"2023-07-14T17:43:44.894007+0000","last_clean":"2023-07-14T17:43:44.894007+0000","last_became_active":"2023-07-14T17:41:43.695511+0000","last_became_peered":"2023-07-14T17:41:43.695511+0000","last_unstale":"2023-07-14T17:43:44.894007+0000","last_undegraded":"2023-07-14T17:43:44.894007+0000","last_fullsized":"2023-07-14T17:43:44.894007+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:38:50.657865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:38:50.657865+0000","last_clean_scrub_stamp":"2023-07-14T17:38:50.657865+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:01:26.054999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.325941+0000","last_change":"2023-07-14T17:43:45.325941+0000","last_active":"2023-07-14T17:43:45.325941+0000","last_peered":"2023-07-14T17:43:45.325941+0000","last_clean":"2023-07-14T17:43:45.325941+0000","last_became_active":"2023-07-14T17:43:45.325519+0000","last_became_peered":"2023-07-14T17:43:45.325519+0000","last_unstale":"2023-07-14T17:43:45.325941+0000","last_undegraded":"2023-07-14T17:43:45.325941+0000","last_fullsized":"2023-07-14T17:43:45.325941+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:06:30.535779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.323905+0000","last_change":"2023-07-14T17:42:42.543243+0000","last_active":"2023-07-14T17:43:45.323905+0000","last_peered":"2023-07-14T17:43:45.323905+0000","last_clean":"2023-07-14T17:43:45.323905+0000","last_became_active":"2023-07-14T17:42:42.542965+0000","last_became_peered":"2023-07-14T17:42:42.542965+0000","last_unstale":"2023-07-14T17:43:45.323905+0000","last_undegraded":"2023-07-14T17:43:45.323905+0000","last_fullsized":"2023-07-14T17:43:45.323905+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:11:31.220902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894165+0000","last_change":"2023-07-14T17:42:41.920806+0000","last_active":"2023-07-14T17:43:44.894165+0000","last_peered":"2023-07-14T17:43:44.894165+0000","last_clean":"2023-07-14T17:43:44.894165+0000","last_became_active":"2023-07-14T17:42:41.920528+0000","last_became_peered":"2023-07-14T17:42:41.920528+0000","last_unstale":"2023-07-14T17:43:44.894165+0000","last_undegraded":"2023-07-14T17:43:44.894165+0000","last_fullsized":"2023-07-14T17:43:44.894165+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:56:57.810566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.891910+0000","last_change":"2023-07-14T17:42:46.542026+0000","last_active":"2023-07-14T17:43:44.891910+0000","last_peered":"2023-07-14T17:43:44.891910+0000","last_clean":"2023-07-14T17:43:44.891910+0000","last_became_active":"2023-07-14T17:42:46.541672+0000","last_became_peered":"2023-07-14T17:42:46.541672+0000","last_unstale":"2023-07-14T17:43:44.891910+0000","last_undegraded":"2023-07-14T17:43:44.891910+0000","last_fullsized":"2023-07-14T17:43:44.891910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:31:32.137890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896620+0000","last_change":"2023-07-14T17:42:41.915798+0000","last_active":"2023-07-14T17:43:44.896620+0000","last_peered":"2023-07-14T17:43:44.896620+0000","last_clean":"2023-07-14T17:43:44.896620+0000","last_became_active":"2023-07-14T17:42:41.915492+0000","last_became_peered":"2023-07-14T17:42:41.915492+0000","last_unstale":"2023-07-14T17:43:44.896620+0000","last_undegraded":"2023-07-14T17:43:44.896620+0000","last_fullsized":"2023-07-14T17:43:44.896620+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:08:22.653303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324401+0000","last_change":"2023-07-14T17:42:42.533500+0000","last_active":"2023-07-14T17:43:45.324401+0000","last_peered":"2023-07-14T17:43:45.324401+0000","last_clean":"2023-07-14T17:43:45.324401+0000","last_became_active":"2023-07-14T17:42:42.533232+0000","last_became_peered":"2023-07-14T17:42:42.533232+0000","last_unstale":"2023-07-14T17:43:45.324401+0000","last_undegraded":"2023-07-14T17:43:45.324401+0000","last_fullsized":"2023-07-14T17:43:45.324401+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:42:49.105557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895003+0000","last_change":"2023-07-14T17:42:46.577131+0000","last_active":"2023-07-14T17:43:44.895003+0000","last_peered":"2023-07-14T17:43:44.895003+0000","last_clean":"2023-07-14T17:43:44.895003+0000","last_became_active":"2023-07-14T17:42:46.576695+0000","last_became_peered":"2023-07-14T17:42:46.576695+0000","last_unstale":"2023-07-14T17:43:44.895003+0000","last_undegraded":"2023-07-14T17:43:44.895003+0000","last_fullsized":"2023-07-14T17:43:44.895003+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:46:57.153931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'9","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.893473+0000","last_change":"2023-07-14T17:42:41.919021+0000","last_active":"2023-07-14T17:43:44.893473+0000","last_peered":"2023-07-14T17:43:44.893473+0000","last_clean":"2023-07-14T17:43:44.893473+0000","last_became_active":"2023-07-14T17:42:41.918714+0000","last_became_peered":"2023-07-14T17:42:41.918714+0000","last_unstale":"2023-07-14T17:43:44.893473+0000","last_undegraded":"2023-07-14T17:43:44.893473+0000","last_fullsized":"2023-07-14T17:43:44.893473+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:11:20.666027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897256+0000","last_change":"2023-07-14T17:42:42.534661+0000","last_active":"2023-07-14T17:43:44.897256+0000","last_peered":"2023-07-14T17:43:44.897256+0000","last_clean":"2023-07-14T17:43:44.897256+0000","last_became_active":"2023-07-14T17:42:42.534357+0000","last_became_peered":"2023-07-14T17:42:42.534357+0000","last_unstale":"2023-07-14T17:43:44.897256+0000","last_undegraded":"2023-07-14T17:43:44.897256+0000","last_fullsized":"2023-07-14T17:43:44.897256+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:13:04.290337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894357+0000","last_change":"2023-07-14T17:42:46.538648+0000","last_active":"2023-07-14T17:43:44.894357+0000","last_peered":"2023-07-14T17:43:44.894357+0000","last_clean":"2023-07-14T17:43:44.894357+0000","last_became_active":"2023-07-14T17:42:46.538225+0000","last_became_peered":"2023-07-14T17:42:46.538225+0000","last_unstale":"2023-07-14T17:43:44.894357+0000","last_undegraded":"2023-07-14T17:43:44.894357+0000","last_fullsized":"2023-07-14T17:43:44.894357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:21:00.195671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.814243+0000","last_change":"2023-07-14T17:42:41.915406+0000","last_active":"2023-07-14T17:43:45.814243+0000","last_peered":"2023-07-14T17:43:45.814243+0000","last_clean":"2023-07-14T17:43:45.814243+0000","last_became_active":"2023-07-14T17:42:41.914967+0000","last_became_peered":"2023-07-14T17:42:41.914967+0000","last_unstale":"2023-07-14T17:43:45.814243+0000","last_undegraded":"2023-07-14T17:43:45.814243+0000","last_fullsized":"2023-07-14T17:43:45.814243+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:38:20.282477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.325475+0000","last_change":"2023-07-14T17:43:45.325475+0000","last_active":"2023-07-14T17:43:45.325475+0000","last_peered":"2023-07-14T17:43:45.325475+0000","last_clean":"2023-07-14T17:43:45.325475+0000","last_became_active":"2023-07-14T17:43:45.325074+0000","last_became_peered":"2023-07-14T17:43:45.325074+0000","last_unstale":"2023-07-14T17:43:45.325475+0000","last_undegraded":"2023-07-14T17:43:45.325475+0000","last_fullsized":"2023-07-14T17:43:45.325475+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:45:33.551379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.814371+0000","last_change":"2023-07-14T17:42:46.577224+0000","last_active":"2023-07-14T17:43:45.814371+0000","last_peered":"2023-07-14T17:43:45.814371+0000","last_clean":"2023-07-14T17:43:45.814371+0000","last_became_active":"2023-07-14T17:42:46.576781+0000","last_became_peered":"2023-07-14T17:42:46.576781+0000","last_unstale":"2023-07-14T17:43:45.814371+0000","last_undegraded":"2023-07-14T17:43:45.814371+0000","last_fullsized":"2023-07-14T17:43:45.814371+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:03:30.154940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896001+0000","last_change":"2023-07-14T17:42:41.916406+0000","last_active":"2023-07-14T17:43:44.896001+0000","last_peered":"2023-07-14T17:43:44.896001+0000","last_clean":"2023-07-14T17:43:44.896001+0000","last_became_active":"2023-07-14T17:42:41.916115+0000","last_became_peered":"2023-07-14T17:42:41.916115+0000","last_unstale":"2023-07-14T17:43:44.896001+0000","last_undegraded":"2023-07-14T17:43:44.896001+0000","last_fullsized":"2023-07-14T17:43:44.896001+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:46:09.237959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894118+0000","last_change":"2023-07-14T17:42:42.529969+0000","last_active":"2023-07-14T17:43:44.894118+0000","last_peered":"2023-07-14T17:43:44.894118+0000","last_clean":"2023-07-14T17:43:44.894118+0000","last_became_active":"2023-07-14T17:42:42.529659+0000","last_became_peered":"2023-07-14T17:42:42.529659+0000","last_unstale":"2023-07-14T17:43:44.894118+0000","last_undegraded":"2023-07-14T17:43:44.894118+0000","last_fullsized":"2023-07-14T17:43:44.894118+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:36:09.137465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.893324+0000","last_change":"2023-07-14T17:42:46.546303+0000","last_active":"2023-07-14T17:43:44.893324+0000","last_peered":"2023-07-14T17:43:44.893324+0000","last_clean":"2023-07-14T17:43:44.893324+0000","last_became_active":"2023-07-14T17:42:46.545947+0000","last_became_peered":"2023-07-14T17:42:46.545947+0000","last_unstale":"2023-07-14T17:43:44.893324+0000","last_undegraded":"2023-07-14T17:43:44.893324+0000","last_fullsized":"2023-07-14T17:43:44.893324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:21:36.193257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895642+0000","last_change":"2023-07-14T17:42:41.916373+0000","last_active":"2023-07-14T17:43:44.895642+0000","last_peered":"2023-07-14T17:43:44.895642+0000","last_clean":"2023-07-14T17:43:44.895642+0000","last_became_active":"2023-07-14T17:42:41.915966+0000","last_became_peered":"2023-07-14T17:42:41.915966+0000","last_unstale":"2023-07-14T17:43:44.895642+0000","last_undegraded":"2023-07-14T17:43:44.895642+0000","last_fullsized":"2023-07-14T17:43:44.895642+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:32:53.280585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.891773+0000","last_change":"2023-07-14T17:42:42.532040+0000","last_active":"2023-07-14T17:43:44.891773+0000","last_peered":"2023-07-14T17:43:44.891773+0000","last_clean":"2023-07-14T17:43:44.891773+0000","last_became_active":"2023-07-14T17:42:42.531690+0000","last_became_peered":"2023-07-14T17:42:42.531690+0000","last_unstale":"2023-07-14T17:43:44.891773+0000","last_undegraded":"2023-07-14T17:43:44.891773+0000","last_fullsized":"2023-07-14T17:43:44.891773+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:19:44.676745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.814020+0000","last_change":"2023-07-14T17:42:46.550166+0000","last_active":"2023-07-14T17:43:45.814020+0000","last_peered":"2023-07-14T17:43:45.814020+0000","last_clean":"2023-07-14T17:43:45.814020+0000","last_became_active":"2023-07-14T17:42:46.549700+0000","last_became_peered":"2023-07-14T17:42:46.549700+0000","last_unstale":"2023-07-14T17:43:45.814020+0000","last_undegraded":"2023-07-14T17:43:45.814020+0000","last_fullsized":"2023-07-14T17:43:45.814020+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:41:53.732450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.911523+0000","last_change":"2023-07-14T17:43:44.911267+0000","last_active":"2023-07-14T17:43:44.911523+0000","last_peered":"2023-07-14T17:43:44.911523+0000","last_clean":"2023-07-14T17:43:44.911523+0000","last_became_active":"2023-07-14T17:43:44.899552+0000","last_became_peered":"2023-07-14T17:43:44.899552+0000","last_unstale":"2023-07-14T17:43:44.911523+0000","last_undegraded":"2023-07-14T17:43:44.911523+0000","last_fullsized":"2023-07-14T17:43:44.911523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:30:32.991380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897748+0000","last_change":"2023-07-14T17:42:42.531789+0000","last_active":"2023-07-14T17:43:44.897748+0000","last_peered":"2023-07-14T17:43:44.897748+0000","last_clean":"2023-07-14T17:43:44.897748+0000","last_became_active":"2023-07-14T17:42:42.531460+0000","last_became_peered":"2023-07-14T17:42:42.531460+0000","last_unstale":"2023-07-14T17:43:44.897748+0000","last_undegraded":"2023-07-14T17:43:44.897748+0000","last_fullsized":"2023-07-14T17:43:44.897748+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:46.112701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.814241+0000","last_change":"2023-07-14T17:42:46.577034+0000","last_active":"2023-07-14T17:43:45.814241+0000","last_peered":"2023-07-14T17:43:45.814241+0000","last_clean":"2023-07-14T17:43:45.814241+0000","last_became_active":"2023-07-14T17:42:46.576636+0000","last_became_peered":"2023-07-14T17:42:46.576636+0000","last_unstale":"2023-07-14T17:43:45.814241+0000","last_undegraded":"2023-07-14T17:43:45.814241+0000","last_fullsized":"2023-07-14T17:43:45.814241+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:40:08.514523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894683+0000","last_change":"2023-07-14T17:42:41.919099+0000","last_active":"2023-07-14T17:43:44.894683+0000","last_peered":"2023-07-14T17:43:44.894683+0000","last_clean":"2023-07-14T17:43:44.894683+0000","last_became_active":"2023-07-14T17:42:41.918306+0000","last_became_peered":"2023-07-14T17:42:41.918306+0000","last_unstale":"2023-07-14T17:43:44.894683+0000","last_undegraded":"2023-07-14T17:43:44.894683+0000","last_fullsized":"2023-07-14T17:43:44.894683+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:38:47.955835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894358+0000","last_change":"2023-07-14T17:42:42.552969+0000","last_active":"2023-07-14T17:43:44.894358+0000","last_peered":"2023-07-14T17:43:44.894358+0000","last_clean":"2023-07-14T17:43:44.894358+0000","last_became_active":"2023-07-14T17:42:42.552647+0000","last_became_peered":"2023-07-14T17:42:42.552647+0000","last_unstale":"2023-07-14T17:43:44.894358+0000","last_undegraded":"2023-07-14T17:43:44.894358+0000","last_fullsized":"2023-07-14T17:43:44.894358+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:41:32.606687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896695+0000","last_change":"2023-07-14T17:42:46.538335+0000","last_active":"2023-07-14T17:43:44.896695+0000","last_peered":"2023-07-14T17:43:44.896695+0000","last_clean":"2023-07-14T17:43:44.896695+0000","last_became_active":"2023-07-14T17:42:46.537929+0000","last_became_peered":"2023-07-14T17:42:46.537929+0000","last_unstale":"2023-07-14T17:43:44.896695+0000","last_undegraded":"2023-07-14T17:43:44.896695+0000","last_fullsized":"2023-07-14T17:43:44.896695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:03:03.278360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":77,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:48.879539+0000","last_change":"2023-07-14T17:42:41.916323+0000","last_active":"2023-07-14T17:43:48.879539+0000","last_peered":"2023-07-14T17:43:48.879539+0000","last_clean":"2023-07-14T17:43:48.879539+0000","last_became_active":"2023-07-14T17:42:41.916019+0000","last_became_peered":"2023-07-14T17:42:41.916019+0000","last_unstale":"2023-07-14T17:43:48.879539+0000","last_undegraded":"2023-07-14T17:43:48.879539+0000","last_fullsized":"2023-07-14T17:43:48.879539+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:29:56.282750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":48,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894518+0000","last_change":"2023-07-14T17:42:42.549843+0000","last_active":"2023-07-14T17:43:44.894518+0000","last_peered":"2023-07-14T17:43:44.894518+0000","last_clean":"2023-07-14T17:43:44.894518+0000","last_became_active":"2023-07-14T17:42:42.549347+0000","last_became_peered":"2023-07-14T17:42:42.549347+0000","last_unstale":"2023-07-14T17:43:44.894518+0000","last_undegraded":"2023-07-14T17:43:44.894518+0000","last_fullsized":"2023-07-14T17:43:44.894518+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:32:23.560638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.898526+0000","last_change":"2023-07-14T17:42:46.545262+0000","last_active":"2023-07-14T17:43:44.898526+0000","last_peered":"2023-07-14T17:43:44.898526+0000","last_clean":"2023-07-14T17:43:44.898526+0000","last_became_active":"2023-07-14T17:42:46.544891+0000","last_became_peered":"2023-07-14T17:42:46.544891+0000","last_unstale":"2023-07-14T17:43:44.898526+0000","last_undegraded":"2023-07-14T17:43:44.898526+0000","last_fullsized":"2023-07-14T17:43:44.898526+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:58:48.061797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897783+0000","last_change":"2023-07-14T17:42:41.912855+0000","last_active":"2023-07-14T17:43:44.897783+0000","last_peered":"2023-07-14T17:43:44.897783+0000","last_clean":"2023-07-14T17:43:44.897783+0000","last_became_active":"2023-07-14T17:42:41.912510+0000","last_became_peered":"2023-07-14T17:42:41.912510+0000","last_unstale":"2023-07-14T17:43:44.897783+0000","last_undegraded":"2023-07-14T17:43:44.897783+0000","last_fullsized":"2023-07-14T17:43:44.897783+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:29:49.003962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894208+0000","last_change":"2023-07-14T17:42:42.545344+0000","last_active":"2023-07-14T17:43:44.894208+0000","last_peered":"2023-07-14T17:43:44.894208+0000","last_clean":"2023-07-14T17:43:44.894208+0000","last_became_active":"2023-07-14T17:42:42.545056+0000","last_became_peered":"2023-07-14T17:42:42.545056+0000","last_unstale":"2023-07-14T17:43:44.894208+0000","last_undegraded":"2023-07-14T17:43:44.894208+0000","last_fullsized":"2023-07-14T17:43:44.894208+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:31:59.187718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897362+0000","last_change":"2023-07-14T17:42:46.541279+0000","last_active":"2023-07-14T17:43:44.897362+0000","last_peered":"2023-07-14T17:43:44.897362+0000","last_clean":"2023-07-14T17:43:44.897362+0000","last_became_active":"2023-07-14T17:42:46.540835+0000","last_became_peered":"2023-07-14T17:42:46.540835+0000","last_unstale":"2023-07-14T17:43:44.897362+0000","last_undegraded":"2023-07-14T17:43:44.897362+0000","last_fullsized":"2023-07-14T17:43:44.897362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:41:42.325967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324578+0000","last_change":"2023-07-14T17:42:41.919257+0000","last_active":"2023-07-14T17:43:45.324578+0000","last_peered":"2023-07-14T17:43:45.324578+0000","last_clean":"2023-07-14T17:43:45.324578+0000","last_became_active":"2023-07-14T17:42:41.918968+0000","last_became_peered":"2023-07-14T17:42:41.918968+0000","last_unstale":"2023-07-14T17:43:45.324578+0000","last_undegraded":"2023-07-14T17:43:45.324578+0000","last_fullsized":"2023-07-14T17:43:45.324578+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:06:24.227953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894456+0000","last_change":"2023-07-14T17:42:42.543993+0000","last_active":"2023-07-14T17:43:44.894456+0000","last_peered":"2023-07-14T17:43:44.894456+0000","last_clean":"2023-07-14T17:43:44.894456+0000","last_became_active":"2023-07-14T17:42:42.543685+0000","last_became_peered":"2023-07-14T17:42:42.543685+0000","last_unstale":"2023-07-14T17:43:44.894456+0000","last_undegraded":"2023-07-14T17:43:44.894456+0000","last_fullsized":"2023-07-14T17:43:44.894456+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:45:26.338845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.891847+0000","last_change":"2023-07-14T17:42:42.538553+0000","last_active":"2023-07-14T17:43:44.891847+0000","last_peered":"2023-07-14T17:43:44.891847+0000","last_clean":"2023-07-14T17:43:44.891847+0000","last_became_active":"2023-07-14T17:42:42.538251+0000","last_became_peered":"2023-07-14T17:42:42.538251+0000","last_unstale":"2023-07-14T17:43:44.891847+0000","last_undegraded":"2023-07-14T17:43:44.891847+0000","last_fullsized":"2023-07-14T17:43:44.891847+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:44:50.672379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894621+0000","last_change":"2023-07-14T17:42:42.543819+0000","last_active":"2023-07-14T17:43:44.894621+0000","last_peered":"2023-07-14T17:43:44.894621+0000","last_clean":"2023-07-14T17:43:44.894621+0000","last_became_active":"2023-07-14T17:42:42.543311+0000","last_became_peered":"2023-07-14T17:42:42.543311+0000","last_unstale":"2023-07-14T17:43:44.894621+0000","last_undegraded":"2023-07-14T17:43:44.894621+0000","last_fullsized":"2023-07-14T17:43:44.894621+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:03:11.132113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894009+0000","last_change":"2023-07-14T17:42:42.537525+0000","last_active":"2023-07-14T17:43:44.894009+0000","last_peered":"2023-07-14T17:43:44.894009+0000","last_clean":"2023-07-14T17:43:44.894009+0000","last_became_active":"2023-07-14T17:42:42.537242+0000","last_became_peered":"2023-07-14T17:42:42.537242+0000","last_unstale":"2023-07-14T17:43:44.894009+0000","last_undegraded":"2023-07-14T17:43:44.894009+0000","last_fullsized":"2023-07-14T17:43:44.894009+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:40:58.077332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.898807+0000","last_change":"2023-07-14T17:42:42.547670+0000","last_active":"2023-07-14T17:43:44.898807+0000","last_peered":"2023-07-14T17:43:44.898807+0000","last_clean":"2023-07-14T17:43:44.898807+0000","last_became_active":"2023-07-14T17:42:42.547349+0000","last_became_peered":"2023-07-14T17:42:42.547349+0000","last_unstale":"2023-07-14T17:43:44.898807+0000","last_undegraded":"2023-07-14T17:43:44.898807+0000","last_fullsized":"2023-07-14T17:43:44.898807+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:49:40.819754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324379+0000","last_change":"2023-07-14T17:42:42.542741+0000","last_active":"2023-07-14T17:43:45.324379+0000","last_peered":"2023-07-14T17:43:45.324379+0000","last_clean":"2023-07-14T17:43:45.324379+0000","last_became_active":"2023-07-14T17:42:42.541972+0000","last_became_peered":"2023-07-14T17:42:42.541972+0000","last_unstale":"2023-07-14T17:43:45.324379+0000","last_undegraded":"2023-07-14T17:43:45.324379+0000","last_fullsized":"2023-07-14T17:43:45.324379+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:27.843013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892205+0000","last_change":"2023-07-14T17:42:42.535043+0000","last_active":"2023-07-14T17:43:44.892205+0000","last_peered":"2023-07-14T17:43:44.892205+0000","last_clean":"2023-07-14T17:43:44.892205+0000","last_became_active":"2023-07-14T17:42:42.534281+0000","last_became_peered":"2023-07-14T17:42:42.534281+0000","last_unstale":"2023-07-14T17:43:44.892205+0000","last_undegraded":"2023-07-14T17:43:44.892205+0000","last_fullsized":"2023-07-14T17:43:44.892205+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:27:45.984466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895460+0000","last_change":"2023-07-14T17:42:42.540866+0000","last_active":"2023-07-14T17:43:44.895460+0000","last_peered":"2023-07-14T17:43:44.895460+0000","last_clean":"2023-07-14T17:43:44.895460+0000","last_became_active":"2023-07-14T17:42:42.540568+0000","last_became_peered":"2023-07-14T17:42:42.540568+0000","last_unstale":"2023-07-14T17:43:44.895460+0000","last_undegraded":"2023-07-14T17:43:44.895460+0000","last_fullsized":"2023-07-14T17:43:44.895460+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:12:01.698385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897437+0000","last_change":"2023-07-14T17:42:42.541646+0000","last_active":"2023-07-14T17:43:44.897437+0000","last_peered":"2023-07-14T17:43:44.897437+0000","last_clean":"2023-07-14T17:43:44.897437+0000","last_became_active":"2023-07-14T17:42:42.541324+0000","last_became_peered":"2023-07-14T17:42:42.541324+0000","last_unstale":"2023-07-14T17:43:44.897437+0000","last_undegraded":"2023-07-14T17:43:44.897437+0000","last_fullsized":"2023-07-14T17:43:44.897437+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:37:25.561305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897528+0000","last_change":"2023-07-14T17:42:42.534650+0000","last_active":"2023-07-14T17:43:44.897528+0000","last_peered":"2023-07-14T17:43:44.897528+0000","last_clean":"2023-07-14T17:43:44.897528+0000","last_became_active":"2023-07-14T17:42:42.534351+0000","last_became_peered":"2023-07-14T17:42:42.534351+0000","last_unstale":"2023-07-14T17:43:44.897528+0000","last_undegraded":"2023-07-14T17:43:44.897528+0000","last_fullsized":"2023-07-14T17:43:44.897528+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:13:52.397350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895517+0000","last_change":"2023-07-14T17:42:42.535815+0000","last_active":"2023-07-14T17:43:44.895517+0000","last_peered":"2023-07-14T17:43:44.895517+0000","last_clean":"2023-07-14T17:43:44.895517+0000","last_became_active":"2023-07-14T17:42:42.535510+0000","last_became_peered":"2023-07-14T17:42:42.535510+0000","last_unstale":"2023-07-14T17:43:44.895517+0000","last_undegraded":"2023-07-14T17:43:44.895517+0000","last_fullsized":"2023-07-14T17:43:44.895517+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:52:40.052652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897009+0000","last_change":"2023-07-14T17:42:42.551758+0000","last_active":"2023-07-14T17:43:44.897009+0000","last_peered":"2023-07-14T17:43:44.897009+0000","last_clean":"2023-07-14T17:43:44.897009+0000","last_became_active":"2023-07-14T17:42:42.551444+0000","last_became_peered":"2023-07-14T17:42:42.551444+0000","last_unstale":"2023-07-14T17:43:44.897009+0000","last_undegraded":"2023-07-14T17:43:44.897009+0000","last_fullsized":"2023-07-14T17:43:44.897009+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:26:22.278980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324115+0000","last_change":"2023-07-14T17:42:42.546355+0000","last_active":"2023-07-14T17:43:45.324115+0000","last_peered":"2023-07-14T17:43:45.324115+0000","last_clean":"2023-07-14T17:43:45.324115+0000","last_became_active":"2023-07-14T17:42:42.546079+0000","last_became_peered":"2023-07-14T17:42:42.546079+0000","last_unstale":"2023-07-14T17:43:45.324115+0000","last_undegraded":"2023-07-14T17:43:45.324115+0000","last_fullsized":"2023-07-14T17:43:45.324115+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:57:16.604025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.901014+0000","last_change":"2023-07-14T17:43:44.901014+0000","last_active":"2023-07-14T17:43:44.901014+0000","last_peered":"2023-07-14T17:43:44.901014+0000","last_clean":"2023-07-14T17:43:44.901014+0000","last_became_active":"2023-07-14T17:43:44.900632+0000","last_became_peered":"2023-07-14T17:43:44.900632+0000","last_unstale":"2023-07-14T17:43:44.901014+0000","last_undegraded":"2023-07-14T17:43:44.901014+0000","last_fullsized":"2023-07-14T17:43:44.901014+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:23:03.050997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65561,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":493,"num_read_kb":317,"num_write":234,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":196683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":206,"ondisk_log_size":206,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481817,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":486248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.53900000000000003}]},{"osd":1,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":2.8330000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.39000000000000001}]},{"osd":2,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.36899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":4,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.44600000000000001}]},{"osd":5,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.499},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.52900000000000003}]},{"osd":6,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.65100000000000002}]},{"osd":7,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.621},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.51800000000000002}]},{"osd":8,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.63},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.64100000000000001}]},{"osd":9,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.65800000000000003}]},{"osd":10,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.68600000000000005}]}]},{"osd":10,"up_from":69,"seq":296352743453,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28672,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716480,"statfs":{"total":95995035648,"available":95965675520,"internally_reserved":0,"allocated":1687552,"data_stored":1132005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.86399999999999999}]},{"osd":1,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.27500000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.77600000000000002,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.48299999999999998,"15min":0.47899999999999998},"min":{"1min":0.33400000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.59499999999999997,"15min":0.58499999999999996},"min":{"1min":0.51800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.83199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.629,"15min":0.627},"min":{"1min":0.48299999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.754}]},{"osd":3,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.627,"15min":0.64300000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69699999999999995,"5min":0.876,"15min":0.876},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84899999999999998,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.69299999999999995}]},{"osd":4,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.61799999999999999,"15min":0.62},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.77500000000000002,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.54200000000000004,"15min":0.54000000000000004},"min":{"1min":0.41499999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.62}]},{"osd":5,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.53200000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.83399999999999996,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.73799999999999999,"15min":0.746},"min":{"1min":0.53800000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.82899999999999996,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.65900000000000003}]},{"osd":6,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.72499999999999998,"15min":0.74099999999999999},"min":{"1min":0.41399999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78800000000000003,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.47299999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.86699999999999999,"5min":1.125,"15min":1.125},"last":0.76900000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":8,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80000000000000004,"15min":0.79900000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.91000000000000003,"5min":1.204,"15min":1.204},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.76900000000000002,"15min":0.77200000000000002},"min":{"1min":0.60299999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88900000000000001,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.89100000000000001}]},{"osd":9,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.82699999999999996,"15min":0.81799999999999995},"min":{"1min":0.65500000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.76200000000000001,"15min":0.77100000000000002},"min":{"1min":0.51300000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.92300000000000004,"5min":1.079,"15min":1.079},"last":0.91500000000000004}]},{"osd":11,"last update":"Fri Jul 14 17:43:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.90300000000000002}]}]},{"osd":9,"up_from":64,"seq":274877906977,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":490322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.41099999999999998,"15min":0.42399999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.497,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.45400000000000001,"15min":0.47199999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.55800000000000005,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.38900000000000001}]},{"osd":1,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.59099999999999997,"15min":0.60599999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.69899999999999995,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.60199999999999998,"15min":0.61599999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70999999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.51100000000000001}]},{"osd":2,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79900000000000004,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.64500000000000002,"15min":0.65700000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.73199999999999998,"5min":1.119,"15min":1.119},"last":0.55000000000000004}]},{"osd":3,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.48999999999999999,"15min":0.50600000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.52700000000000002,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.496},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.49199999999999999,"15min":0.505},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64400000000000002,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.57299999999999995}]},{"osd":4,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.628,"15min":0.64300000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.70699999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.63600000000000001,"15min":0.64800000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.68300000000000005,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.66800000000000004}]},{"osd":5,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.64600000000000002,"15min":0.65800000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.78400000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.65800000000000003,"15min":0.67100000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79600000000000004,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.63200000000000001}]},{"osd":6,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.372},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.52500000000000002}]},{"osd":7,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.70599999999999996,"15min":0.72799999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71099999999999997,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.71699999999999997,"15min":0.73799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.79400000000000004,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.66100000000000003}]},{"osd":8,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.749,"15min":0.76800000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.82699999999999996,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.621},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.74199999999999999,"15min":0.75800000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.85899999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.70099999999999996}]},{"osd":10,"last update":"Fri Jul 14 17:43:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.73399999999999999,"15min":0.755},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.74399999999999999,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.73199999999999998,"15min":0.748},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.73999999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.72599999999999998}]},{"osd":11,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.75},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.68200000000000005}]}]},{"osd":8,"up_from":57,"seq":244813135910,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":486604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53400000000000003,"15min":0.53300000000000003},"min":{"1min":0.35499999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.45800000000000002}]},{"osd":1,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.372,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83899999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.40999999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.55100000000000005}]},{"osd":2,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45800000000000002,"15min":0.46100000000000002},"min":{"1min":0.27500000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60599999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.435},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.47199999999999998,"15min":0.47999999999999998},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56499999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.503}]},{"osd":3,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.36299999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.56699999999999995}]},{"osd":4,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85099999999999998,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64300000000000002,"15min":0.64200000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89700000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.61499999999999999}]},{"osd":5,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57399999999999995,"15min":0.58099999999999996},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72499999999999998,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.312,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.75,"5min":1.018,"15min":1.018},"last":0.63900000000000001}]},{"osd":6,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.66600000000000004,"15min":0.66800000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68500000000000005,"15min":0.69299999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84799999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.63100000000000001}]},{"osd":7,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.51000000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92400000000000004,"5min":1.008,"15min":1.008},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71099999999999997,"15min":0.70999999999999996},"min":{"1min":0.48899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88200000000000001,"5min":1.258,"15min":1.258},"last":0.66900000000000004}]},{"osd":9,"last update":"Fri Jul 14 17:43:19 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.68500000000000005,"15min":0.67000000000000004},"min":{"1min":0.441,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.68600000000000005,"15min":0.67200000000000004},"min":{"1min":0.46899999999999997,"5min":0.44,"15min":0.44},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.71899999999999997}]},{"osd":10,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.69699999999999995,"15min":0.68100000000000005},"min":{"1min":0.57199999999999995,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94399999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.70699999999999996,"15min":0.68500000000000005},"min":{"1min":0.61099999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.73699999999999999}]},{"osd":11,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.69999999999999996}]}]},{"osd":7,"up_from":49,"seq":210453397546,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1613824,"data_stored":1083346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.503},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.59499999999999997}]},{"osd":1,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.42199999999999999,"15min":0.42499999999999999},"min":{"1min":0.29999999999999999,"5min":0.25,"15min":0.25},"max":{"1min":0.70299999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.47299999999999998},"min":{"1min":0.32400000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69499999999999995,"5min":0.746,"15min":0.746},"last":0.497}]},{"osd":2,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.64300000000000002,"15min":0.60799999999999998},"min":{"1min":0.443,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82799999999999996,"5min":5.1050000000000004,"15min":5.1050000000000004},"last":0.622},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.59799999999999998,"15min":0.58599999999999997},"min":{"1min":0.47099999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.83699999999999997,"5min":1.048,"15min":1.048},"last":0.58899999999999997}]},{"osd":3,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.41799999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86499999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.60899999999999999},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80700000000000005,"5min":0.995,"15min":0.995},"last":0.66100000000000003}]},{"osd":4,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.505,"15min":0.503},"min":{"1min":0.36799999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.76300000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52000000000000002,"15min":0.53200000000000003},"min":{"1min":0.307,"5min":0.248,"15min":0.248},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.54400000000000004}]},{"osd":5,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.64800000000000002,"15min":0.63600000000000001},"min":{"1min":0.499,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98099999999999998,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.65600000000000003,"15min":0.64800000000000002},"min":{"1min":0.51600000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.96499999999999997,"5min":1.02,"15min":1.02},"last":0.67500000000000004}]},{"osd":6,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.64200000000000002,"15min":0.65000000000000002},"min":{"1min":0.45200000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.878,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.67600000000000005,"15min":0.68700000000000006},"min":{"1min":0.46700000000000003,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.94899999999999995,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.73599999999999999}]},{"osd":8,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.45700000000000002,"5min":0.441,"15min":0.441},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.71299999999999997,"15min":0.72099999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0620000000000001,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.68899999999999995}]},{"osd":9,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.69399999999999995,"15min":0.70199999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.01,"5min":1.196,"15min":1.196},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.91200000000000003,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.70299999999999996}]},{"osd":10,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.65800000000000003,"15min":0.63700000000000001},"min":{"1min":0.53400000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.67700000000000005,"15min":0.66100000000000003},"min":{"1min":0.505,"5min":0.441,"15min":0.441},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.753}]},{"osd":11,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.82799999999999996}]}]},{"osd":6,"up_from":44,"seq":188978561070,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":481682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.36799999999999999,"15min":0.35499999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.28199999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.28199999999999997}]},{"osd":1,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57999999999999996,"15min":0.56799999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.76000000000000001,"5min":0.87,"15min":0.87},"last":0.503},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.65000000000000002,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.371}]},{"osd":2,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59399999999999997,"15min":0.58999999999999997},"min":{"1min":0.44500000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.53600000000000003}]},{"osd":3,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.44400000000000001,"15min":0.44700000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68500000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.312},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.47199999999999998,"15min":0.45400000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78900000000000003,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.40400000000000003}]},{"osd":4,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85299999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.60799999999999998,"15min":0.60599999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81799999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.45000000000000001}]},{"osd":5,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.60099999999999998,"15min":0.59299999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.68600000000000005,"15min":0.66300000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.8679999999999999,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.56000000000000005}]},{"osd":7,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.754,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.66600000000000004,"15min":0.68000000000000005},"min":{"1min":0.47999999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.73499999999999999,"5min":1.012,"15min":1.012},"last":0.47699999999999998}]},{"osd":8,"last update":"Fri Jul 14 17:43:04 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.73599999999999999,"15min":0.755},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.98999999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.69499999999999995,"15min":0.71099999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92500000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.46300000000000002}]},{"osd":9,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.68100000000000005,"15min":0.70099999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.91500000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.62},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.65700000000000003,"15min":0.67700000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.90200000000000002,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.48899999999999999}]},{"osd":10,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.72799999999999998,"15min":0.746},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.81999999999999995,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.755,"15min":0.76900000000000002},"min":{"1min":0.52500000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.83799999999999997,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.60099999999999998}]},{"osd":11,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.63600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":541263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.503,"15min":0.46700000000000003},"min":{"1min":0.33800000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.37},{"interface":"front","average":{"1min":0.497,"5min":0.51500000000000001,"15min":0.47199999999999998},"min":{"1min":0.33600000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.78600000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.55100000000000005}]},{"osd":2,"last update":"Fri Jul 14 17:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.54200000000000004,"15min":0.497},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.86099999999999999,"5min":0.879,"15min":0.879},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.52300000000000002,"15min":0.48899999999999999},"min":{"1min":0.36499999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.59599999999999997}]},{"osd":3,"last update":"Fri Jul 14 17:43:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53400000000000003,"15min":0.51600000000000001},"min":{"1min":0.31,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.75600000000000001,"5min":0.873,"15min":0.873},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57699999999999996,"15min":0.54500000000000004},"min":{"1min":0.44700000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.752,"5min":0.873,"15min":0.873},"last":0.42399999999999999}]},{"osd":4,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.45400000000000001,"15min":0.42999999999999999},"min":{"1min":0.34699999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.45700000000000002,"15min":0.433},"min":{"1min":0.29399999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.32600000000000001}]},{"osd":5,"last update":"Fri Jul 14 17:42:59 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.60899999999999999,"15min":0.58499999999999996},"min":{"1min":0.48299999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.58199999999999996,"15min":0.57099999999999995},"min":{"1min":0.42699999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.65500000000000003}]},{"osd":6,"last update":"Fri Jul 14 17:43:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.41599999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.438}]},{"osd":7,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.56899999999999995,"15min":0.52800000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.55300000000000005,"15min":0.53300000000000003},"min":{"1min":0.373,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.46200000000000002}]},{"osd":8,"last update":"Fri Jul 14 17:42:59 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.65900000000000003,"15min":0.65600000000000003},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1,"5min":1,"15min":1},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.54000000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.66500000000000004}]},{"osd":9,"last update":"Fri Jul 14 17:43:19 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65500000000000003,"15min":0.65700000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.88400000000000001,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70199999999999996,"15min":0.70099999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.499}]},{"osd":10,"last update":"Fri Jul 14 17:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.747,"15min":0.73699999999999999},"min":{"1min":0.52600000000000002,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.71399999999999997,"15min":0.70099999999999996},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.51200000000000001}]},{"osd":11,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.69899999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":483138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 17:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.61499999999999999,"15min":0.53700000000000003},"min":{"1min":0.38500000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":8.5199999999999996,"5min":8.5199999999999996,"15min":8.5199999999999996},"last":0.5},{"interface":"front","average":{"1min":1.0960000000000001,"5min":0.65800000000000003,"15min":0.56000000000000005},"min":{"1min":0.41599999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":10.31,"5min":10.31,"15min":10.31},"last":0.441}]},{"osd":2,"last update":"Fri Jul 14 17:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55900000000000005,"15min":0.53400000000000003},"min":{"1min":0.40999999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56999999999999995,"15min":0.55300000000000005},"min":{"1min":0.379,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.55500000000000005}]},{"osd":3,"last update":"Fri Jul 14 17:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":8.4629999999999992,"5min":8.4629999999999992,"15min":8.4629999999999992},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":8.4900000000000002,"5min":8.4900000000000002,"15min":8.4900000000000002},"last":0.52800000000000002}]},{"osd":4,"last update":"Fri Jul 14 17:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55700000000000005,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.121,"5min":0.68100000000000005,"15min":0.58099999999999996},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":10.374000000000001,"5min":10.374000000000001,"15min":10.374000000000001},"last":0.57199999999999995}]},{"osd":5,"last update":"Fri Jul 14 17:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56499999999999995,"15min":0.55100000000000005},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.755,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59599999999999997,"15min":0.58999999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84099999999999997,"5min":0.88,"15min":0.88},"last":0.61899999999999999}]},{"osd":6,"last update":"Fri Jul 14 17:43:18 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.57999999999999996,"15min":0.502},"min":{"1min":0.30599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":10.468999999999999,"5min":10.468999999999999,"15min":10.468999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.58499999999999996,"15min":0.51600000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":8.4350000000000005,"5min":8.4350000000000005,"15min":8.4350000000000005},"last":0.64300000000000002}]},{"osd":7,"last update":"Fri Jul 14 17:43:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.65300000000000002}]},{"osd":8,"last update":"Fri Jul 14 17:43:06 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.68500000000000005,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.82799999999999996,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.67100000000000004,"15min":0.67900000000000005},"min":{"1min":0.46200000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.89900000000000002,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.69299999999999995}]},{"osd":9,"last update":"Fri Jul 14 17:43:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.77600000000000002,"15min":0.72799999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":10.343,"5min":10.343,"15min":10.343},"last":0.63500000000000001},{"interface":"front","average":{"1min":1.5509999999999999,"5min":0.89000000000000001,"15min":0.78000000000000003},"min":{"1min":0.45300000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":18.739000000000001,"5min":18.739000000000001,"15min":18.739000000000001},"last":0.68500000000000005}]},{"osd":10,"last update":"Fri Jul 14 17:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.73299999999999998,"15min":0.74199999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.755},{"interface":"front","average":{"1min":1.2090000000000001,"5min":0.85999999999999999,"15min":0.80200000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":10.439,"5min":10.439,"15min":10.439},"last":0.67600000000000005}]},{"osd":11,"last update":"Fri Jul 14 17:43:06 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.72799999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":541552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.47499999999999998},"min":{"1min":0.41099999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.48299999999999998,"15min":0.46700000000000003},"min":{"1min":0.39100000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.48099999999999998}]},{"osd":1,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.53100000000000003,"15min":0.48299999999999998},"min":{"1min":0.35199999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.55300000000000005,"15min":0.50900000000000001},"min":{"1min":0.5,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.93300000000000005}]},{"osd":3,"last update":"Fri Jul 14 17:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.57499999999999996,"15min":0.54700000000000004},"min":{"1min":0.38100000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55900000000000005,"15min":0.53900000000000003},"min":{"1min":0.40100000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.86899999999999999,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.54500000000000004}]},{"osd":4,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.82699999999999996}]},{"osd":5,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.45600000000000002,"15min":0.43099999999999999},"min":{"1min":0.30399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.45200000000000001,"15min":0.437},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.45900000000000002}]},{"osd":6,"last update":"Fri Jul 14 17:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.60099999999999998,"15min":0.58099999999999996},"min":{"1min":0.47199999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.76000000000000001,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59899999999999998,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81799999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.65200000000000002}]},{"osd":7,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.96299999999999997}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.61299999999999999,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.59299999999999997,"15min":0.57399999999999995},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.63300000000000001}]},{"osd":9,"last update":"Fri Jul 14 17:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71399999999999997,"15min":0.70999999999999996},"min":{"1min":0.56999999999999995,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.752},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.76700000000000002,"15min":0.75600000000000001},"min":{"1min":0.64100000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.85099999999999998}]},{"osd":10,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.74099999999999999,"15min":0.73699999999999999},"min":{"1min":0.51300000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.77500000000000002,"15min":0.76300000000000001},"min":{"1min":0.59299999999999997,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.876}]},{"osd":11,"last update":"Fri Jul 14 17:43:40 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.91500000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":481438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:16 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34300000000000003,"15min":0.32300000000000001},"min":{"1min":0.26900000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.437,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.36099999999999999,"15min":0.33200000000000002},"min":{"1min":0.28100000000000003,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.307}]},{"osd":1,"last update":"Fri Jul 14 17:43:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54200000000000004,"15min":0.50600000000000001},"min":{"1min":0.38600000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.82599999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54200000000000004,"15min":0.505},"min":{"1min":0.33300000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72499999999999998,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.34000000000000002}]},{"osd":2,"last update":"Fri Jul 14 17:43:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54700000000000004,"15min":0.52400000000000002},"min":{"1min":0.41899999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.69699999999999995,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.52800000000000002},"min":{"1min":0.40899999999999997,"5min":0.379,"15min":0.379},"max":{"1min":0.71999999999999997,"5min":1.423,"15min":1.423},"last":0.52800000000000002}]},{"osd":4,"last update":"Fri Jul 14 17:43:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58299999999999996,"15min":0.56699999999999995},"min":{"1min":0.378,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.748,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.44},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61199999999999999,"15min":0.58499999999999996},"min":{"1min":0.42499999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79400000000000004,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.57099999999999995}]},{"osd":5,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.38100000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85299999999999998,"5min":1.593,"15min":1.593},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.45400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.85699999999999998,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.59499999999999997}]},{"osd":6,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.52600000000000002,"15min":0.52900000000000003},"min":{"1min":0.32800000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.73699999999999999,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.46899999999999997,"15min":0.47599999999999998},"min":{"1min":0.32700000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.57399999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.54400000000000004}]},{"osd":7,"last update":"Fri Jul 14 17:43:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.61899999999999999,"15min":0.63700000000000001},"min":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82999999999999996,"5min":1.28,"15min":1.28},"last":0.505},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65500000000000003,"15min":0.64500000000000002},"min":{"1min":0.54400000000000004,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.82299999999999995,"5min":1.03,"15min":1.03},"last":0.63200000000000001}]},{"osd":8,"last update":"Fri Jul 14 17:41:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.55800000000000005}]},{"osd":9,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.878,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.68100000000000005,"15min":0.69699999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.60999999999999999}]},{"osd":10,"last update":"Fri Jul 14 17:43:44 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66200000000000003,"15min":0.67500000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.67700000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.749,"15min":0.76000000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.86799999999999999,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.66100000000000003}]},{"osd":11,"last update":"Fri Jul 14 17:43:44 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.67500000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953526,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":486146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.35399999999999998,"5min":0.311,"15min":0.311},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.52100000000000002,"15min":0.502},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.496}]},{"osd":1,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.41699999999999998,"15min":0.38200000000000001},"min":{"1min":0.372,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.61699999999999999,"5min":0.755,"15min":0.755},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.42699999999999999,"15min":0.40000000000000002},"min":{"1min":0.38,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.621,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.44400000000000001}]},{"osd":2,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.57499999999999996,"15min":0.56299999999999994},"min":{"1min":0.496,"5min":0.38,"15min":0.38},"max":{"1min":0.80200000000000005,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.622,"15min":0.58199999999999996},"min":{"1min":0.54400000000000004,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89600000000000002,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.497}]},{"osd":3,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.59799999999999998,"15min":0.55100000000000005},"min":{"1min":0.44600000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.86599999999999999,"5min":1.498,"15min":1.498},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56200000000000006,"15min":0.53400000000000003},"min":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81100000000000005,"5min":1.629,"15min":1.629},"last":0.48399999999999999}]},{"osd":5,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.63100000000000001,"15min":0.61099999999999999},"min":{"1min":0.51500000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89800000000000002,"5min":1.256,"15min":1.256},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.624,"15min":0.60999999999999999},"min":{"1min":0.44900000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.59199999999999997}]},{"osd":6,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.72199999999999998,"15min":0.746},"min":{"1min":0.47699999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91800000000000004,"5min":2.7789999999999999,"15min":2.7789999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68200000000000005,"15min":0.66700000000000004},"min":{"1min":0.46600000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.95899999999999996,"5min":1.476,"15min":1.476},"last":0.54900000000000004}]},{"osd":7,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.57899999999999996,"15min":0.53300000000000003},"min":{"1min":0.49099999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.97299999999999998,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.61399999999999999,"15min":0.58399999999999996},"min":{"1min":0.5,"5min":0.377,"15min":0.377},"max":{"1min":0.88500000000000001,"5min":1.002,"15min":1.002},"last":0.62}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.65600000000000003,"15min":0.64400000000000002},"min":{"1min":0.41999999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72299999999999998,"15min":0.71399999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.64800000000000002}]},{"osd":9,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.69399999999999995,"15min":0.69099999999999995},"min":{"1min":0.54000000000000004,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88100000000000001,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.80100000000000005,"15min":0.80400000000000005},"min":{"1min":0.61899999999999999,"5min":0.502,"15min":0.502},"max":{"1min":1.0049999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.60499999999999998}]},{"osd":10,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.82899999999999996,"15min":0.83099999999999996},"min":{"1min":0.63400000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0600000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.73199999999999998,"15min":0.72999999999999998},"min":{"1min":0.59799999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.63200000000000001}]},{"osd":11,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.69299999999999995}]}]},{"osd":5,"up_from":37,"seq":158913790003,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1622016,"data_stored":1078634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47099999999999997,"15min":0.44700000000000001},"min":{"1min":0.35499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77800000000000002,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.501,"5min":0.48299999999999998,"15min":0.45700000000000002},"min":{"1min":0.35599999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65400000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.64900000000000002}]},{"osd":1,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.55100000000000005,"15min":0.497},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":5.3630000000000004,"5min":5.3630000000000004,"15min":5.3630000000000004},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51300000000000001,"15min":0.48999999999999999},"min":{"1min":0.42399999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.40799999999999997}]},{"osd":2,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.33500000000000002},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.376,"15min":0.35999999999999999},"min":{"1min":0.30499999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.33500000000000002}]},{"osd":3,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.501,"15min":0.495},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.51800000000000002,"15min":0.495},"min":{"1min":0.38700000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.747,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.72199999999999998}]},{"osd":4,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.53500000000000003,"15min":0.50800000000000001},"min":{"1min":0.434,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78100000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.435},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54300000000000004,"15min":0.52000000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.38500000000000001}]},{"osd":6,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.39000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.75700000000000001,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.56699999999999995,"15min":0.54300000000000004},"min":{"1min":0.46400000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80800000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.68600000000000005}]},{"osd":7,"last update":"Fri Jul 14 17:43:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.58899999999999997,"15min":0.57399999999999995},"min":{"1min":0.47599999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.63300000000000001,"15min":0.61299999999999999},"min":{"1min":0.51200000000000001,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.52100000000000002}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.54700000000000004,"15min":0.54000000000000004},"min":{"1min":0.38100000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.754,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.372,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.876,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.41899999999999998}]},{"osd":9,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.56999999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91000000000000003,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68400000000000005,"15min":0.67600000000000005},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.61199999999999999}]},{"osd":10,"last update":"Fri Jul 14 17:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.69199999999999995,"15min":0.68200000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.88200000000000001,"15min":0.89400000000000002},"min":{"1min":0.623,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":0.94599999999999995,"5min":5.3330000000000002,"15min":5.3330000000000002},"last":0.623}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":60629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T17:43:55.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:55 smithi178 ceph-mon[129368]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:43:55.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:55 smithi178 ceph-mon[129368]: pgmap v293: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 0 B/s, 0 keys/s, 1 objects/s recovering 2023-07-14T17:43:55.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:55 smithi088 ceph-mon[127465]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:43:55.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:55 smithi088 ceph-mon[127465]: pgmap v293: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 0 B/s, 0 keys/s, 1 objects/s recovering 2023-07-14T17:43:55.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:55 smithi107 ceph-mon[128999]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:43:55.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:55 smithi107 ceph-mon[128999]: pgmap v293: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s; 0 B/s, 0 keys/s, 1 objects/s recovering 2023-07-14T17:43:56.686 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T17:43:57.163 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:56 smithi178 ceph-mon[129368]: pgmap v294: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s; 0 B/s, 0 keys/s, 1 objects/s recovering 2023-07-14T17:43:57.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:56 smithi088 ceph-mon[127465]: pgmap v294: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s; 0 B/s, 0 keys/s, 1 objects/s recovering 2023-07-14T17:43:57.223 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:56 smithi107 ceph-mon[128999]: pgmap v294: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 12 op/s; 0 B/s, 0 keys/s, 1 objects/s recovering 2023-07-14T17:43:57.253 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:43:58.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.614+0000 7faab96f7700 1 -- 172.21.15.88:0/970164455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faab4109620 msgr2=0x7faab410ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:58.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.614+0000 7faab96f7700 1 --2- 172.21.15.88:0/970164455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faab4109620 0x7faab410ba00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7faaac00b0a0 tx=0x7faaac00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:43:58.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.614+0000 7faab96f7700 1 -- 172.21.15.88:0/970164455 shutdown_connections 2023-07-14T17:43:58.596 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.614+0000 7faab96f7700 1 --2- 172.21.15.88:0/970164455 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faab4109620 0x7faab410ba00 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.614+0000 7faab96f7700 1 --2- 172.21.15.88:0/970164455 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab4100070 0x7faab4100440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.614+0000 7faab96f7700 1 --2- 172.21.15.88:0/970164455 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faab4100980 0x7faab41090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.614+0000 7faab96f7700 1 -- 172.21.15.88:0/970164455 >> 172.21.15.88:0/970164455 conn(0x7faab4074550 msgr2=0x7faab4074950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.614+0000 7faab96f7700 1 -- 172.21.15.88:0/970164455 shutdown_connections 2023-07-14T17:43:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.614+0000 7faab96f7700 1 -- 172.21.15.88:0/970164455 wait complete. 2023-07-14T17:43:58.597 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.614+0000 7faab96f7700 1 Processor -- start 2023-07-14T17:43:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.615+0000 7faab96f7700 1 -- start start 2023-07-14T17:43:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.615+0000 7faab96f7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab4100070 0x7faab4195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.615+0000 7faab96f7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faab4100980 0x7faab4196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.615+0000 7faab96f7700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faab4109620 0x7faab419a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.615+0000 7faab96f7700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab419adf0 con 0x7faab4100980 2023-07-14T17:43:58.599 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.615+0000 7faab96f7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab419af60 con 0x7faab4100070 2023-07-14T17:43:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.615+0000 7faab96f7700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab419b120 con 0x7faab4109620 2023-07-14T17:43:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.615+0000 7faab27fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faab4100980 0x7faab4196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.615+0000 7faab27fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faab4100980 0x7faab4196370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42488/0 (socket says 172.21.15.88:42488) 2023-07-14T17:43:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.615+0000 7faab27fc700 1 -- 172.21.15.88:0/1722273119 learned_addr learned my addr 172.21.15.88:0/1722273119 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:43:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.615+0000 7faab2ffd700 1 --2- 172.21.15.88:0/1722273119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab4100070 0x7faab4195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.615+0000 7faab37fe700 1 --2- 172.21.15.88:0/1722273119 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faab4109620 0x7faab419a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:58.600 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.615+0000 7faab27fc700 1 -- 172.21.15.88:0/1722273119 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faab4109620 msgr2=0x7faab419a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:58.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.615+0000 7faab27fc700 1 --2- 172.21.15.88:0/1722273119 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faab4109620 0x7faab419a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:58.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.615+0000 7faab27fc700 1 -- 172.21.15.88:0/1722273119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab4100070 msgr2=0x7faab4195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:58.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.615+0000 7faab27fc700 1 --2- 172.21.15.88:0/1722273119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab4100070 0x7faab4195e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:58.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.615+0000 7faab27fc700 1 -- 172.21.15.88:0/1722273119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaac009d00 con 0x7faab4100980 2023-07-14T17:43:58.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.616+0000 7faab27fc700 1 --2- 172.21.15.88:0/1722273119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faab4100980 0x7faab4196370 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7faaa400b530 tx=0x7faaa400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:58.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.616+0000 7faa9bfff700 1 -- 172.21.15.88:0/1722273119 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa4011830 con 0x7faab4100980 2023-07-14T17:43:58.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.616+0000 7faab96f7700 1 -- 172.21.15.88:0/1722273119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab419b400 con 0x7faab4100980 2023-07-14T17:43:58.601 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.616+0000 7faa9bfff700 1 -- 172.21.15.88:0/1722273119 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faaa4011e20 con 0x7faab4100980 2023-07-14T17:43:58.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.616+0000 7faa9bfff700 1 -- 172.21.15.88:0/1722273119 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa4010db0 con 0x7faab4100980 2023-07-14T17:43:58.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.616+0000 7faab96f7700 1 -- 172.21.15.88:0/1722273119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab41a2c60 con 0x7faab4100980 2023-07-14T17:43:58.602 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.617+0000 7faab96f7700 1 -- 172.21.15.88:0/1722273119 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab404e340 con 0x7faab4100980 2023-07-14T17:43:58.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.618+0000 7faa9bfff700 1 -- 172.21.15.88:0/1722273119 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7faaa4011990 con 0x7faab4100980 2023-07-14T17:43:58.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.618+0000 7faa9bfff700 1 --2- 172.21.15.88:0/1722273119 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7faa9c0a11e0 0x7faa9c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:43:58.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.618+0000 7faab2ffd700 1 --2- 172.21.15.88:0/1722273119 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7faa9c0a11e0 0x7faa9c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:43:58.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.618+0000 7faa9bfff700 1 -- 172.21.15.88:0/1722273119 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7faaa40cc670 con 0x7faab4100980 2023-07-14T17:43:58.603 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.619+0000 7faab2ffd700 1 --2- 172.21.15.88:0/1722273119 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7faa9c0a11e0 0x7faa9c0a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7faaa8009c60 tx=0x7faaa8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:43:58.604 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.622+0000 7faa9bfff700 1 -- 172.21.15.88:0/1722273119 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faaa4095a60 con 0x7faab4100980 2023-07-14T17:43:58.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.806+0000 7faab96f7700 1 -- 172.21.15.88:0/1722273119 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faab40fe900 con 0x7faa9c0a11e0 2023-07-14T17:43:58.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.818+0000 7faa9bfff700 1 -- 172.21.15.88:0/1722273119 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427177 (secure 0 0 0) 0x7faab40fe900 con 0x7faa9c0a11e0 2023-07-14T17:43:58.803 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:43:58.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.821+0000 7faab96f7700 1 -- 172.21.15.88:0/1722273119 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7faa9c0a11e0 msgr2=0x7faa9c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:58.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.821+0000 7faab96f7700 1 --2- 172.21.15.88:0/1722273119 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7faa9c0a11e0 0x7faa9c0a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7faaa8009c60 tx=0x7faaa8009260 comp rx=0 tx=0).stop 2023-07-14T17:43:58.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.821+0000 7faab96f7700 1 -- 172.21.15.88:0/1722273119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faab4100980 msgr2=0x7faab4196370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:43:58.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.821+0000 7faab96f7700 1 --2- 172.21.15.88:0/1722273119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faab4100980 0x7faab4196370 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7faaa400b530 tx=0x7faaa400b840 comp rx=0 tx=0).stop 2023-07-14T17:43:58.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.821+0000 7faab96f7700 1 -- 172.21.15.88:0/1722273119 shutdown_connections 2023-07-14T17:43:58.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.821+0000 7faab96f7700 1 --2- 172.21.15.88:0/1722273119 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7faab4109620 0x7faab419a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:58.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.821+0000 7faab96f7700 1 --2- 172.21.15.88:0/1722273119 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7faab4100070 0x7faab4195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:58.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.821+0000 7faab96f7700 1 --2- 172.21.15.88:0/1722273119 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7faa9c0a11e0 0x7faa9c0a3690 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:58.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.821+0000 7faab96f7700 1 --2- 172.21.15.88:0/1722273119 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faab4100980 0x7faab4196370 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:43:58.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.821+0000 7faab96f7700 1 -- 172.21.15.88:0/1722273119 >> 172.21.15.88:0/1722273119 conn(0x7faab4074550 msgr2=0x7faab40fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:43:58.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.821+0000 7faab96f7700 1 -- 172.21.15.88:0/1722273119 shutdown_connections 2023-07-14T17:43:58.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:43:58.821+0000 7faab96f7700 1 -- 172.21.15.88:0/1722273119 wait complete. 2023-07-14T17:43:58.808 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2023-07-14T17:43:59.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:58 smithi088 ceph-mon[127465]: pgmap v295: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2023-07-14T17:43:59.215 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2023-07-14T17:43:57.890702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656175,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":669,"num_read_kb":451,"num_write":417,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1180736,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":345,"ondisk_log_size":345,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":338068,"kb_used_data":13792,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124603756,"statfs":{"total":1151940427776,"available":1151594246144,"internally_reserved":0,"allocated":14123008,"data_stored":7772378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003392"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892404+0000","last_change":"2023-07-14T17:42:42.540396+0000","last_active":"2023-07-14T17:43:44.892404+0000","last_peered":"2023-07-14T17:43:44.892404+0000","last_clean":"2023-07-14T17:43:44.892404+0000","last_became_active":"2023-07-14T17:42:42.539732+0000","last_became_peered":"2023-07-14T17:42:42.539732+0000","last_unstale":"2023-07-14T17:43:44.892404+0000","last_undegraded":"2023-07-14T17:43:44.892404+0000","last_fullsized":"2023-07-14T17:43:44.892404+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:45:35.673679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.899803+0000","last_change":"2023-07-14T17:42:42.532153+0000","last_active":"2023-07-14T17:43:44.899803+0000","last_peered":"2023-07-14T17:43:44.899803+0000","last_clean":"2023-07-14T17:43:44.899803+0000","last_became_active":"2023-07-14T17:42:42.531745+0000","last_became_peered":"2023-07-14T17:42:42.531745+0000","last_unstale":"2023-07-14T17:43:44.899803+0000","last_undegraded":"2023-07-14T17:43:44.899803+0000","last_fullsized":"2023-07-14T17:43:44.899803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:38:17.037985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.898523+0000","last_change":"2023-07-14T17:42:42.549497+0000","last_active":"2023-07-14T17:43:44.898523+0000","last_peered":"2023-07-14T17:43:44.898523+0000","last_clean":"2023-07-14T17:43:44.898523+0000","last_became_active":"2023-07-14T17:42:42.549201+0000","last_became_peered":"2023-07-14T17:42:42.549201+0000","last_unstale":"2023-07-14T17:43:44.898523+0000","last_undegraded":"2023-07-14T17:43:44.898523+0000","last_fullsized":"2023-07-14T17:43:44.898523+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:07:07.859324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.814335+0000","last_change":"2023-07-14T17:42:42.543460+0000","last_active":"2023-07-14T17:43:45.814335+0000","last_peered":"2023-07-14T17:43:45.814335+0000","last_clean":"2023-07-14T17:43:45.814335+0000","last_became_active":"2023-07-14T17:42:42.543127+0000","last_became_peered":"2023-07-14T17:42:42.543127+0000","last_unstale":"2023-07-14T17:43:45.814335+0000","last_undegraded":"2023-07-14T17:43:45.814335+0000","last_fullsized":"2023-07-14T17:43:45.814335+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:06:06.644157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894474+0000","last_change":"2023-07-14T17:42:42.534242+0000","last_active":"2023-07-14T17:43:44.894474+0000","last_peered":"2023-07-14T17:43:44.894474+0000","last_clean":"2023-07-14T17:43:44.894474+0000","last_became_active":"2023-07-14T17:42:42.533536+0000","last_became_peered":"2023-07-14T17:42:42.533536+0000","last_unstale":"2023-07-14T17:43:44.894474+0000","last_undegraded":"2023-07-14T17:43:44.894474+0000","last_fullsized":"2023-07-14T17:43:44.894474+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:53:30.057794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896541+0000","last_change":"2023-07-14T17:42:42.538042+0000","last_active":"2023-07-14T17:43:44.896541+0000","last_peered":"2023-07-14T17:43:44.896541+0000","last_clean":"2023-07-14T17:43:44.896541+0000","last_became_active":"2023-07-14T17:42:42.537612+0000","last_became_peered":"2023-07-14T17:42:42.537612+0000","last_unstale":"2023-07-14T17:43:44.896541+0000","last_undegraded":"2023-07-14T17:43:44.896541+0000","last_fullsized":"2023-07-14T17:43:44.896541+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:29:37.412187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894322+0000","last_change":"2023-07-14T17:42:42.545318+0000","last_active":"2023-07-14T17:43:44.894322+0000","last_peered":"2023-07-14T17:43:44.894322+0000","last_clean":"2023-07-14T17:43:44.894322+0000","last_became_active":"2023-07-14T17:42:42.544984+0000","last_became_peered":"2023-07-14T17:42:42.544984+0000","last_unstale":"2023-07-14T17:43:44.894322+0000","last_undegraded":"2023-07-14T17:43:44.894322+0000","last_fullsized":"2023-07-14T17:43:44.894322+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:10:59.223910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.900595+0000","last_change":"2023-07-14T17:43:44.899856+0000","last_active":"2023-07-14T17:43:44.900595+0000","last_peered":"2023-07-14T17:43:44.900595+0000","last_clean":"2023-07-14T17:43:44.900595+0000","last_became_active":"2023-07-14T17:43:44.899478+0000","last_became_peered":"2023-07-14T17:43:44.899478+0000","last_unstale":"2023-07-14T17:43:44.900595+0000","last_undegraded":"2023-07-14T17:43:44.900595+0000","last_fullsized":"2023-07-14T17:43:44.900595+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:32:02.556140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897879+0000","last_change":"2023-07-14T17:42:42.548973+0000","last_active":"2023-07-14T17:43:44.897879+0000","last_peered":"2023-07-14T17:43:44.897879+0000","last_clean":"2023-07-14T17:43:44.897879+0000","last_became_active":"2023-07-14T17:42:42.548672+0000","last_became_peered":"2023-07-14T17:42:42.548672+0000","last_unstale":"2023-07-14T17:43:44.897879+0000","last_undegraded":"2023-07-14T17:43:44.897879+0000","last_fullsized":"2023-07-14T17:43:44.897879+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:55:26.825661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896875+0000","last_change":"2023-07-14T17:42:42.540969+0000","last_active":"2023-07-14T17:43:44.896875+0000","last_peered":"2023-07-14T17:43:44.896875+0000","last_clean":"2023-07-14T17:43:44.896875+0000","last_became_active":"2023-07-14T17:42:42.540653+0000","last_became_peered":"2023-07-14T17:42:42.540653+0000","last_unstale":"2023-07-14T17:43:44.896875+0000","last_undegraded":"2023-07-14T17:43:44.896875+0000","last_fullsized":"2023-07-14T17:43:44.896875+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:07.062764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894491+0000","last_change":"2023-07-14T17:42:42.550016+0000","last_active":"2023-07-14T17:43:44.894491+0000","last_peered":"2023-07-14T17:43:44.894491+0000","last_clean":"2023-07-14T17:43:44.894491+0000","last_became_active":"2023-07-14T17:42:42.549653+0000","last_became_peered":"2023-07-14T17:42:42.549653+0000","last_unstale":"2023-07-14T17:43:44.894491+0000","last_undegraded":"2023-07-14T17:43:44.894491+0000","last_fullsized":"2023-07-14T17:43:44.894491+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:43.271416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.814147+0000","last_change":"2023-07-14T17:42:42.552105+0000","last_active":"2023-07-14T17:43:45.814147+0000","last_peered":"2023-07-14T17:43:45.814147+0000","last_clean":"2023-07-14T17:43:45.814147+0000","last_became_active":"2023-07-14T17:42:42.550830+0000","last_became_peered":"2023-07-14T17:42:42.550830+0000","last_unstale":"2023-07-14T17:43:45.814147+0000","last_undegraded":"2023-07-14T17:43:45.814147+0000","last_fullsized":"2023-07-14T17:43:45.814147+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:08:48.231339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324361+0000","last_change":"2023-07-14T17:42:42.534963+0000","last_active":"2023-07-14T17:43:45.324361+0000","last_peered":"2023-07-14T17:43:45.324361+0000","last_clean":"2023-07-14T17:43:45.324361+0000","last_became_active":"2023-07-14T17:42:42.534546+0000","last_became_peered":"2023-07-14T17:42:42.534546+0000","last_unstale":"2023-07-14T17:43:45.324361+0000","last_undegraded":"2023-07-14T17:43:45.324361+0000","last_fullsized":"2023-07-14T17:43:45.324361+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:37:07.860214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.898319+0000","last_change":"2023-07-14T17:42:42.530073+0000","last_active":"2023-07-14T17:43:44.898319+0000","last_peered":"2023-07-14T17:43:44.898319+0000","last_clean":"2023-07-14T17:43:44.898319+0000","last_became_active":"2023-07-14T17:42:42.529767+0000","last_became_peered":"2023-07-14T17:42:42.529767+0000","last_unstale":"2023-07-14T17:43:44.898319+0000","last_undegraded":"2023-07-14T17:43:44.898319+0000","last_fullsized":"2023-07-14T17:43:44.898319+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:07:57.028886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.898063+0000","last_change":"2023-07-14T17:42:42.548168+0000","last_active":"2023-07-14T17:43:44.898063+0000","last_peered":"2023-07-14T17:43:44.898063+0000","last_clean":"2023-07-14T17:43:44.898063+0000","last_became_active":"2023-07-14T17:42:42.547363+0000","last_became_peered":"2023-07-14T17:42:42.547363+0000","last_unstale":"2023-07-14T17:43:44.898063+0000","last_undegraded":"2023-07-14T17:43:44.898063+0000","last_fullsized":"2023-07-14T17:43:44.898063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:43:45.670787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895946+0000","last_change":"2023-07-14T17:42:42.551042+0000","last_active":"2023-07-14T17:43:44.895946+0000","last_peered":"2023-07-14T17:43:44.895946+0000","last_clean":"2023-07-14T17:43:44.895946+0000","last_became_active":"2023-07-14T17:42:42.550730+0000","last_became_peered":"2023-07-14T17:42:42.550730+0000","last_unstale":"2023-07-14T17:43:44.895946+0000","last_undegraded":"2023-07-14T17:43:44.895946+0000","last_fullsized":"2023-07-14T17:43:44.895946+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:33:43.890169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892180+0000","last_change":"2023-07-14T17:42:42.540528+0000","last_active":"2023-07-14T17:43:44.892180+0000","last_peered":"2023-07-14T17:43:44.892180+0000","last_clean":"2023-07-14T17:43:44.892180+0000","last_became_active":"2023-07-14T17:42:42.540093+0000","last_became_peered":"2023-07-14T17:42:42.540093+0000","last_unstale":"2023-07-14T17:43:44.892180+0000","last_undegraded":"2023-07-14T17:43:44.892180+0000","last_fullsized":"2023-07-14T17:43:44.892180+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:13:34.468585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.893607+0000","last_change":"2023-07-14T17:42:42.539512+0000","last_active":"2023-07-14T17:43:44.893607+0000","last_peered":"2023-07-14T17:43:44.893607+0000","last_clean":"2023-07-14T17:43:44.893607+0000","last_became_active":"2023-07-14T17:42:42.539248+0000","last_became_peered":"2023-07-14T17:42:42.539248+0000","last_unstale":"2023-07-14T17:43:44.893607+0000","last_undegraded":"2023-07-14T17:43:44.893607+0000","last_fullsized":"2023-07-14T17:43:44.893607+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:32:15.064855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895813+0000","last_change":"2023-07-14T17:42:42.531630+0000","last_active":"2023-07-14T17:43:44.895813+0000","last_peered":"2023-07-14T17:43:44.895813+0000","last_clean":"2023-07-14T17:43:44.895813+0000","last_became_active":"2023-07-14T17:42:42.531333+0000","last_became_peered":"2023-07-14T17:42:42.531333+0000","last_unstale":"2023-07-14T17:43:44.895813+0000","last_undegraded":"2023-07-14T17:43:44.895813+0000","last_fullsized":"2023-07-14T17:43:44.895813+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:37:35.706668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894477+0000","last_change":"2023-07-14T17:42:46.543062+0000","last_active":"2023-07-14T17:43:44.894477+0000","last_peered":"2023-07-14T17:43:44.894477+0000","last_clean":"2023-07-14T17:43:44.894477+0000","last_became_active":"2023-07-14T17:42:46.542189+0000","last_became_peered":"2023-07-14T17:42:46.542189+0000","last_unstale":"2023-07-14T17:43:44.894477+0000","last_undegraded":"2023-07-14T17:43:44.894477+0000","last_fullsized":"2023-07-14T17:43:44.894477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:57:03.881795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":89,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:51.865876+0000","last_change":"2023-07-14T17:42:41.914589+0000","last_active":"2023-07-14T17:43:51.865876+0000","last_peered":"2023-07-14T17:43:51.865876+0000","last_clean":"2023-07-14T17:43:51.865876+0000","last_became_active":"2023-07-14T17:42:41.914275+0000","last_became_peered":"2023-07-14T17:42:41.914275+0000","last_unstale":"2023-07-14T17:43:51.865876+0000","last_undegraded":"2023-07-14T17:43:51.865876+0000","last_fullsized":"2023-07-14T17:43:51.865876+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:16.066793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":50,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.900792+0000","last_change":"2023-07-14T17:43:44.900792+0000","last_active":"2023-07-14T17:43:44.900792+0000","last_peered":"2023-07-14T17:43:44.900792+0000","last_clean":"2023-07-14T17:43:44.900792+0000","last_became_active":"2023-07-14T17:43:44.900333+0000","last_became_peered":"2023-07-14T17:43:44.900333+0000","last_unstale":"2023-07-14T17:43:44.900792+0000","last_undegraded":"2023-07-14T17:43:44.900792+0000","last_fullsized":"2023-07-14T17:43:44.900792+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:29:20.451913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,8],"acting":[3,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.813589+0000","last_change":"2023-07-14T17:42:46.542500+0000","last_active":"2023-07-14T17:43:45.813589+0000","last_peered":"2023-07-14T17:43:45.813589+0000","last_clean":"2023-07-14T17:43:45.813589+0000","last_became_active":"2023-07-14T17:42:46.541932+0000","last_became_peered":"2023-07-14T17:42:46.541932+0000","last_unstale":"2023-07-14T17:43:45.813589+0000","last_undegraded":"2023-07-14T17:43:45.813589+0000","last_fullsized":"2023-07-14T17:43:45.813589+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:49:41.767435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897058+0000","last_change":"2023-07-14T17:42:41.918169+0000","last_active":"2023-07-14T17:43:44.897058+0000","last_peered":"2023-07-14T17:43:44.897058+0000","last_clean":"2023-07-14T17:43:44.897058+0000","last_became_active":"2023-07-14T17:42:41.917124+0000","last_became_peered":"2023-07-14T17:42:41.917124+0000","last_unstale":"2023-07-14T17:43:44.897058+0000","last_undegraded":"2023-07-14T17:43:44.897058+0000","last_fullsized":"2023-07-14T17:43:44.897058+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:05:43.117229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.898688+0000","last_change":"2023-07-14T17:42:42.539265+0000","last_active":"2023-07-14T17:43:44.898688+0000","last_peered":"2023-07-14T17:43:44.898688+0000","last_clean":"2023-07-14T17:43:44.898688+0000","last_became_active":"2023-07-14T17:42:42.538933+0000","last_became_peered":"2023-07-14T17:42:42.538933+0000","last_unstale":"2023-07-14T17:43:44.898688+0000","last_undegraded":"2023-07-14T17:43:44.898688+0000","last_fullsized":"2023-07-14T17:43:44.898688+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:30:34.884083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.813943+0000","last_change":"2023-07-14T17:42:46.550194+0000","last_active":"2023-07-14T17:43:45.813943+0000","last_peered":"2023-07-14T17:43:45.813943+0000","last_clean":"2023-07-14T17:43:45.813943+0000","last_became_active":"2023-07-14T17:42:46.549693+0000","last_became_peered":"2023-07-14T17:42:46.549693+0000","last_unstale":"2023-07-14T17:43:45.813943+0000","last_undegraded":"2023-07-14T17:43:45.813943+0000","last_fullsized":"2023-07-14T17:43:45.813943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:03:33.891030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.908529+0000","last_change":"2023-07-14T17:43:44.908253+0000","last_active":"2023-07-14T17:43:44.908529+0000","last_peered":"2023-07-14T17:43:44.908529+0000","last_clean":"2023-07-14T17:43:44.908529+0000","last_became_active":"2023-07-14T17:43:44.896490+0000","last_became_peered":"2023-07-14T17:43:44.896490+0000","last_unstale":"2023-07-14T17:43:44.908529+0000","last_undegraded":"2023-07-14T17:43:44.908529+0000","last_fullsized":"2023-07-14T17:43:44.908529+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:21:55.326452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896660+0000","last_change":"2023-07-14T17:42:42.543846+0000","last_active":"2023-07-14T17:43:44.896660+0000","last_peered":"2023-07-14T17:43:44.896660+0000","last_clean":"2023-07-14T17:43:44.896660+0000","last_became_active":"2023-07-14T17:42:42.543528+0000","last_became_peered":"2023-07-14T17:42:42.543528+0000","last_unstale":"2023-07-14T17:43:44.896660+0000","last_undegraded":"2023-07-14T17:43:44.896660+0000","last_fullsized":"2023-07-14T17:43:44.896660+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:14:43.217768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.813751+0000","last_change":"2023-07-14T17:42:46.547799+0000","last_active":"2023-07-14T17:43:45.813751+0000","last_peered":"2023-07-14T17:43:45.813751+0000","last_clean":"2023-07-14T17:43:45.813751+0000","last_became_active":"2023-07-14T17:42:46.547381+0000","last_became_peered":"2023-07-14T17:42:46.547381+0000","last_unstale":"2023-07-14T17:43:45.813751+0000","last_undegraded":"2023-07-14T17:43:45.813751+0000","last_fullsized":"2023-07-14T17:43:45.813751+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:06:35.931658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895253+0000","last_change":"2023-07-14T17:42:41.914911+0000","last_active":"2023-07-14T17:43:44.895253+0000","last_peered":"2023-07-14T17:43:44.895253+0000","last_clean":"2023-07-14T17:43:44.895253+0000","last_became_active":"2023-07-14T17:42:41.914607+0000","last_became_peered":"2023-07-14T17:42:41.914607+0000","last_unstale":"2023-07-14T17:43:44.895253+0000","last_undegraded":"2023-07-14T17:43:44.895253+0000","last_fullsized":"2023-07-14T17:43:44.895253+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:34:12.929058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897636+0000","last_change":"2023-07-14T17:42:42.548866+0000","last_active":"2023-07-14T17:43:44.897636+0000","last_peered":"2023-07-14T17:43:44.897636+0000","last_clean":"2023-07-14T17:43:44.897636+0000","last_became_active":"2023-07-14T17:42:42.548562+0000","last_became_peered":"2023-07-14T17:42:42.548562+0000","last_unstale":"2023-07-14T17:43:44.897636+0000","last_undegraded":"2023-07-14T17:43:44.897636+0000","last_fullsized":"2023-07-14T17:43:44.897636+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:41:33.093669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895655+0000","last_change":"2023-07-14T17:42:46.538879+0000","last_active":"2023-07-14T17:43:44.895655+0000","last_peered":"2023-07-14T17:43:44.895655+0000","last_clean":"2023-07-14T17:43:44.895655+0000","last_became_active":"2023-07-14T17:42:46.538476+0000","last_became_peered":"2023-07-14T17:42:46.538476+0000","last_unstale":"2023-07-14T17:43:44.895655+0000","last_undegraded":"2023-07-14T17:43:44.895655+0000","last_fullsized":"2023-07-14T17:43:44.895655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:50:18.995823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":85,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:53.856883+0000","last_change":"2023-07-14T17:42:41.917215+0000","last_active":"2023-07-14T17:43:53.856883+0000","last_peered":"2023-07-14T17:43:53.856883+0000","last_clean":"2023-07-14T17:43:53.856883+0000","last_became_active":"2023-07-14T17:42:41.916640+0000","last_became_peered":"2023-07-14T17:42:41.916640+0000","last_unstale":"2023-07-14T17:43:53.856883+0000","last_undegraded":"2023-07-14T17:43:53.856883+0000","last_fullsized":"2023-07-14T17:43:53.856883+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:57:44.899889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":53,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897271+0000","last_change":"2023-07-14T17:42:42.534347+0000","last_active":"2023-07-14T17:43:44.897271+0000","last_peered":"2023-07-14T17:43:44.897271+0000","last_clean":"2023-07-14T17:43:44.897271+0000","last_became_active":"2023-07-14T17:42:42.533958+0000","last_became_peered":"2023-07-14T17:42:42.533958+0000","last_unstale":"2023-07-14T17:43:44.897271+0000","last_undegraded":"2023-07-14T17:43:44.897271+0000","last_fullsized":"2023-07-14T17:43:44.897271+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:35:41.285730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894270+0000","last_change":"2023-07-14T17:42:46.543144+0000","last_active":"2023-07-14T17:43:44.894270+0000","last_peered":"2023-07-14T17:43:44.894270+0000","last_clean":"2023-07-14T17:43:44.894270+0000","last_became_active":"2023-07-14T17:42:46.542311+0000","last_became_peered":"2023-07-14T17:42:46.542311+0000","last_unstale":"2023-07-14T17:43:44.894270+0000","last_undegraded":"2023-07-14T17:43:44.894270+0000","last_fullsized":"2023-07-14T17:43:44.894270+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:21:37.953172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896529+0000","last_change":"2023-07-14T17:42:41.916174+0000","last_active":"2023-07-14T17:43:44.896529+0000","last_peered":"2023-07-14T17:43:44.896529+0000","last_clean":"2023-07-14T17:43:44.896529+0000","last_became_active":"2023-07-14T17:42:41.915867+0000","last_became_peered":"2023-07-14T17:42:41.915867+0000","last_unstale":"2023-07-14T17:43:44.896529+0000","last_undegraded":"2023-07-14T17:43:44.896529+0000","last_fullsized":"2023-07-14T17:43:44.896529+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:59:36.014596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897789+0000","last_change":"2023-07-14T17:42:42.547064+0000","last_active":"2023-07-14T17:43:44.897789+0000","last_peered":"2023-07-14T17:43:44.897789+0000","last_clean":"2023-07-14T17:43:44.897789+0000","last_became_active":"2023-07-14T17:42:42.546777+0000","last_became_peered":"2023-07-14T17:42:42.546777+0000","last_unstale":"2023-07-14T17:43:44.897789+0000","last_undegraded":"2023-07-14T17:43:44.897789+0000","last_fullsized":"2023-07-14T17:43:44.897789+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:44:45.781755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892451+0000","last_change":"2023-07-14T17:42:46.548647+0000","last_active":"2023-07-14T17:43:44.892451+0000","last_peered":"2023-07-14T17:43:44.892451+0000","last_clean":"2023-07-14T17:43:44.892451+0000","last_became_active":"2023-07-14T17:42:46.548142+0000","last_became_peered":"2023-07-14T17:42:46.548142+0000","last_unstale":"2023-07-14T17:43:44.892451+0000","last_undegraded":"2023-07-14T17:43:44.892451+0000","last_fullsized":"2023-07-14T17:43:44.892451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:50:46.460689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324145+0000","last_change":"2023-07-14T17:42:41.911675+0000","last_active":"2023-07-14T17:43:45.324145+0000","last_peered":"2023-07-14T17:43:45.324145+0000","last_clean":"2023-07-14T17:43:45.324145+0000","last_became_active":"2023-07-14T17:42:41.911299+0000","last_became_peered":"2023-07-14T17:42:41.911299+0000","last_unstale":"2023-07-14T17:43:45.324145+0000","last_undegraded":"2023-07-14T17:43:45.324145+0000","last_fullsized":"2023-07-14T17:43:45.324145+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:37:15.112003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895160+0000","last_change":"2023-07-14T17:42:42.536211+0000","last_active":"2023-07-14T17:43:44.895160+0000","last_peered":"2023-07-14T17:43:44.895160+0000","last_clean":"2023-07-14T17:43:44.895160+0000","last_became_active":"2023-07-14T17:42:42.535763+0000","last_became_peered":"2023-07-14T17:42:42.535763+0000","last_unstale":"2023-07-14T17:43:44.895160+0000","last_undegraded":"2023-07-14T17:43:44.895160+0000","last_fullsized":"2023-07-14T17:43:44.895160+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:14:23.840550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.899792+0000","last_change":"2023-07-14T17:42:46.540917+0000","last_active":"2023-07-14T17:43:44.899792+0000","last_peered":"2023-07-14T17:43:44.899792+0000","last_clean":"2023-07-14T17:43:44.899792+0000","last_became_active":"2023-07-14T17:42:46.540575+0000","last_became_peered":"2023-07-14T17:42:46.540575+0000","last_unstale":"2023-07-14T17:43:44.899792+0000","last_undegraded":"2023-07-14T17:43:44.899792+0000","last_fullsized":"2023-07-14T17:43:44.899792+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:01:47.539703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897945+0000","last_change":"2023-07-14T17:42:41.917646+0000","last_active":"2023-07-14T17:43:44.897945+0000","last_peered":"2023-07-14T17:43:44.897945+0000","last_clean":"2023-07-14T17:43:44.897945+0000","last_became_active":"2023-07-14T17:42:41.917331+0000","last_became_peered":"2023-07-14T17:42:41.917331+0000","last_unstale":"2023-07-14T17:43:44.897945+0000","last_undegraded":"2023-07-14T17:43:44.897945+0000","last_fullsized":"2023-07-14T17:43:44.897945+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:52:16.441394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892768+0000","last_change":"2023-07-14T17:42:42.526696+0000","last_active":"2023-07-14T17:43:44.892768+0000","last_peered":"2023-07-14T17:43:44.892768+0000","last_clean":"2023-07-14T17:43:44.892768+0000","last_became_active":"2023-07-14T17:42:42.526353+0000","last_became_peered":"2023-07-14T17:42:42.526353+0000","last_unstale":"2023-07-14T17:43:44.892768+0000","last_undegraded":"2023-07-14T17:43:44.892768+0000","last_fullsized":"2023-07-14T17:43:44.892768+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:07:24.895596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895396+0000","last_change":"2023-07-14T17:42:46.545460+0000","last_active":"2023-07-14T17:43:44.895396+0000","last_peered":"2023-07-14T17:43:44.895396+0000","last_clean":"2023-07-14T17:43:44.895396+0000","last_became_active":"2023-07-14T17:42:46.544897+0000","last_became_peered":"2023-07-14T17:42:46.544897+0000","last_unstale":"2023-07-14T17:43:44.895396+0000","last_undegraded":"2023-07-14T17:43:44.895396+0000","last_fullsized":"2023-07-14T17:43:44.895396+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:59:41.696162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":132,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:54.857649+0000","last_change":"2023-07-14T17:42:41.912266+0000","last_active":"2023-07-14T17:43:54.857649+0000","last_peered":"2023-07-14T17:43:54.857649+0000","last_clean":"2023-07-14T17:43:54.857649+0000","last_became_active":"2023-07-14T17:42:41.911944+0000","last_became_peered":"2023-07-14T17:42:41.911944+0000","last_unstale":"2023-07-14T17:43:54.857649+0000","last_undegraded":"2023-07-14T17:43:54.857649+0000","last_fullsized":"2023-07-14T17:43:54.857649+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:17:02.788197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":102,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897607+0000","last_change":"2023-07-14T17:42:42.548609+0000","last_active":"2023-07-14T17:43:44.897607+0000","last_peered":"2023-07-14T17:43:44.897607+0000","last_clean":"2023-07-14T17:43:44.897607+0000","last_became_active":"2023-07-14T17:42:42.548253+0000","last_became_peered":"2023-07-14T17:42:42.548253+0000","last_unstale":"2023-07-14T17:43:44.897607+0000","last_undegraded":"2023-07-14T17:43:44.897607+0000","last_fullsized":"2023-07-14T17:43:44.897607+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:21:18.974575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892477+0000","last_change":"2023-07-14T17:42:46.547324+0000","last_active":"2023-07-14T17:43:44.892477+0000","last_peered":"2023-07-14T17:43:44.892477+0000","last_clean":"2023-07-14T17:43:44.892477+0000","last_became_active":"2023-07-14T17:42:46.546876+0000","last_became_peered":"2023-07-14T17:42:46.546876+0000","last_unstale":"2023-07-14T17:43:44.892477+0000","last_undegraded":"2023-07-14T17:43:44.892477+0000","last_fullsized":"2023-07-14T17:43:44.892477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:07:48.996897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.818146+0000","last_change":"2023-07-14T17:43:45.818146+0000","last_active":"2023-07-14T17:43:45.818146+0000","last_peered":"2023-07-14T17:43:45.818146+0000","last_clean":"2023-07-14T17:43:45.818146+0000","last_became_active":"2023-07-14T17:43:45.814954+0000","last_became_peered":"2023-07-14T17:43:45.814954+0000","last_unstale":"2023-07-14T17:43:45.818146+0000","last_undegraded":"2023-07-14T17:43:45.818146+0000","last_fullsized":"2023-07-14T17:43:45.818146+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:50:05.262289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.898727+0000","last_change":"2023-07-14T17:42:42.548081+0000","last_active":"2023-07-14T17:43:44.898727+0000","last_peered":"2023-07-14T17:43:44.898727+0000","last_clean":"2023-07-14T17:43:44.898727+0000","last_became_active":"2023-07-14T17:42:42.547876+0000","last_became_peered":"2023-07-14T17:42:42.547876+0000","last_unstale":"2023-07-14T17:43:44.898727+0000","last_undegraded":"2023-07-14T17:43:44.898727+0000","last_fullsized":"2023-07-14T17:43:44.898727+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:11:34.698640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.814077+0000","last_change":"2023-07-14T17:42:46.550516+0000","last_active":"2023-07-14T17:43:45.814077+0000","last_peered":"2023-07-14T17:43:45.814077+0000","last_clean":"2023-07-14T17:43:45.814077+0000","last_became_active":"2023-07-14T17:42:46.549588+0000","last_became_peered":"2023-07-14T17:42:46.549588+0000","last_unstale":"2023-07-14T17:43:45.814077+0000","last_undegraded":"2023-07-14T17:43:45.814077+0000","last_fullsized":"2023-07-14T17:43:45.814077+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:55:54.065571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897608+0000","last_change":"2023-07-14T17:42:41.917527+0000","last_active":"2023-07-14T17:43:44.897608+0000","last_peered":"2023-07-14T17:43:44.897608+0000","last_clean":"2023-07-14T17:43:44.897608+0000","last_became_active":"2023-07-14T17:42:41.917239+0000","last_became_peered":"2023-07-14T17:42:41.917239+0000","last_unstale":"2023-07-14T17:43:44.897608+0000","last_undegraded":"2023-07-14T17:43:44.897608+0000","last_fullsized":"2023-07-14T17:43:44.897608+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:07:11.390441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896841+0000","last_change":"2023-07-14T17:42:42.554866+0000","last_active":"2023-07-14T17:43:44.896841+0000","last_peered":"2023-07-14T17:43:44.896841+0000","last_clean":"2023-07-14T17:43:44.896841+0000","last_became_active":"2023-07-14T17:42:42.554585+0000","last_became_peered":"2023-07-14T17:42:42.554585+0000","last_unstale":"2023-07-14T17:43:44.896841+0000","last_undegraded":"2023-07-14T17:43:44.896841+0000","last_fullsized":"2023-07-14T17:43:44.896841+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:38:59.633305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892595+0000","last_change":"2023-07-14T17:42:46.544788+0000","last_active":"2023-07-14T17:43:44.892595+0000","last_peered":"2023-07-14T17:43:44.892595+0000","last_clean":"2023-07-14T17:43:44.892595+0000","last_became_active":"2023-07-14T17:42:46.544419+0000","last_became_peered":"2023-07-14T17:42:46.544419+0000","last_unstale":"2023-07-14T17:43:44.892595+0000","last_undegraded":"2023-07-14T17:43:44.892595+0000","last_fullsized":"2023-07-14T17:43:44.892595+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:26:19.151208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"86'33","reported_seq":167,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:54.857710+0000","last_change":"2023-07-14T17:42:41.917069+0000","last_active":"2023-07-14T17:43:54.857710+0000","last_peered":"2023-07-14T17:43:54.857710+0000","last_clean":"2023-07-14T17:43:54.857710+0000","last_became_active":"2023-07-14T17:42:41.916630+0000","last_became_peered":"2023-07-14T17:42:41.916630+0000","last_unstale":"2023-07-14T17:43:54.857710+0000","last_undegraded":"2023-07-14T17:43:54.857710+0000","last_fullsized":"2023-07-14T17:43:54.857710+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:52:18.568627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58551,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":108,"num_read_kb":110,"num_write":35,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896482+0000","last_change":"2023-07-14T17:42:42.542124+0000","last_active":"2023-07-14T17:43:44.896482+0000","last_peered":"2023-07-14T17:43:44.896482+0000","last_clean":"2023-07-14T17:43:44.896482+0000","last_became_active":"2023-07-14T17:42:42.541851+0000","last_became_peered":"2023-07-14T17:42:42.541851+0000","last_unstale":"2023-07-14T17:43:44.896482+0000","last_undegraded":"2023-07-14T17:43:44.896482+0000","last_fullsized":"2023-07-14T17:43:44.896482+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:56:52.567761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897932+0000","last_change":"2023-07-14T17:42:46.546167+0000","last_active":"2023-07-14T17:43:44.897932+0000","last_peered":"2023-07-14T17:43:44.897932+0000","last_clean":"2023-07-14T17:43:44.897932+0000","last_became_active":"2023-07-14T17:42:46.545768+0000","last_became_peered":"2023-07-14T17:42:46.545768+0000","last_unstale":"2023-07-14T17:43:44.897932+0000","last_undegraded":"2023-07-14T17:43:44.897932+0000","last_fullsized":"2023-07-14T17:43:44.897932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:13:37.105161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894811+0000","last_change":"2023-07-14T17:42:41.918858+0000","last_active":"2023-07-14T17:43:44.894811+0000","last_peered":"2023-07-14T17:43:44.894811+0000","last_clean":"2023-07-14T17:43:44.894811+0000","last_became_active":"2023-07-14T17:42:41.918568+0000","last_became_peered":"2023-07-14T17:42:41.918568+0000","last_unstale":"2023-07-14T17:43:44.894811+0000","last_undegraded":"2023-07-14T17:43:44.894811+0000","last_fullsized":"2023-07-14T17:43:44.894811+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:43:09.338997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324334+0000","last_change":"2023-07-14T17:42:42.542310+0000","last_active":"2023-07-14T17:43:45.324334+0000","last_peered":"2023-07-14T17:43:45.324334+0000","last_clean":"2023-07-14T17:43:45.324334+0000","last_became_active":"2023-07-14T17:42:42.541989+0000","last_became_peered":"2023-07-14T17:42:42.541989+0000","last_unstale":"2023-07-14T17:43:45.324334+0000","last_undegraded":"2023-07-14T17:43:45.324334+0000","last_fullsized":"2023-07-14T17:43:45.324334+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:23:26.638546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.325836+0000","last_change":"2023-07-14T17:43:45.325836+0000","last_active":"2023-07-14T17:43:45.325836+0000","last_peered":"2023-07-14T17:43:45.325836+0000","last_clean":"2023-07-14T17:43:45.325836+0000","last_became_active":"2023-07-14T17:43:45.325330+0000","last_became_peered":"2023-07-14T17:43:45.325330+0000","last_unstale":"2023-07-14T17:43:45.325836+0000","last_undegraded":"2023-07-14T17:43:45.325836+0000","last_fullsized":"2023-07-14T17:43:45.325836+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:34:58.970503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897638+0000","last_change":"2023-07-14T17:42:41.916382+0000","last_active":"2023-07-14T17:43:44.897638+0000","last_peered":"2023-07-14T17:43:44.897638+0000","last_clean":"2023-07-14T17:43:44.897638+0000","last_became_active":"2023-07-14T17:42:41.916082+0000","last_became_peered":"2023-07-14T17:42:41.916082+0000","last_unstale":"2023-07-14T17:43:44.897638+0000","last_undegraded":"2023-07-14T17:43:44.897638+0000","last_fullsized":"2023-07-14T17:43:44.897638+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:28:56.714038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894470+0000","last_change":"2023-07-14T17:42:42.546812+0000","last_active":"2023-07-14T17:43:44.894470+0000","last_peered":"2023-07-14T17:43:44.894470+0000","last_clean":"2023-07-14T17:43:44.894470+0000","last_became_active":"2023-07-14T17:42:42.546517+0000","last_became_peered":"2023-07-14T17:42:42.546517+0000","last_unstale":"2023-07-14T17:43:44.894470+0000","last_undegraded":"2023-07-14T17:43:44.894470+0000","last_fullsized":"2023-07-14T17:43:44.894470+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:01:05.671027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.893117+0000","last_change":"2023-07-14T17:42:46.547726+0000","last_active":"2023-07-14T17:43:44.893117+0000","last_peered":"2023-07-14T17:43:44.893117+0000","last_clean":"2023-07-14T17:43:44.893117+0000","last_became_active":"2023-07-14T17:42:46.547207+0000","last_became_peered":"2023-07-14T17:42:46.547207+0000","last_unstale":"2023-07-14T17:43:44.893117+0000","last_undegraded":"2023-07-14T17:43:44.893117+0000","last_fullsized":"2023-07-14T17:43:44.893117+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:17:09.830803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"86'6","reported_seq":83,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:51.866897+0000","last_change":"2023-07-14T17:42:41.913802+0000","last_active":"2023-07-14T17:43:51.866897+0000","last_peered":"2023-07-14T17:43:51.866897+0000","last_clean":"2023-07-14T17:43:51.866897+0000","last_became_active":"2023-07-14T17:42:41.913467+0000","last_became_peered":"2023-07-14T17:42:41.913467+0000","last_unstale":"2023-07-14T17:43:51.866897+0000","last_undegraded":"2023-07-14T17:43:51.866897+0000","last_fullsized":"2023-07-14T17:43:51.866897+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:28:23.910132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":52,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894173+0000","last_change":"2023-07-14T17:42:42.545379+0000","last_active":"2023-07-14T17:43:44.894173+0000","last_peered":"2023-07-14T17:43:44.894173+0000","last_clean":"2023-07-14T17:43:44.894173+0000","last_became_active":"2023-07-14T17:42:42.545012+0000","last_became_peered":"2023-07-14T17:42:42.545012+0000","last_unstale":"2023-07-14T17:43:44.894173+0000","last_undegraded":"2023-07-14T17:43:44.894173+0000","last_fullsized":"2023-07-14T17:43:44.894173+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:53:26.889969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324438+0000","last_change":"2023-07-14T17:42:46.544524+0000","last_active":"2023-07-14T17:43:45.324438+0000","last_peered":"2023-07-14T17:43:45.324438+0000","last_clean":"2023-07-14T17:43:45.324438+0000","last_became_active":"2023-07-14T17:42:46.544152+0000","last_became_peered":"2023-07-14T17:42:46.544152+0000","last_unstale":"2023-07-14T17:43:45.324438+0000","last_undegraded":"2023-07-14T17:43:45.324438+0000","last_fullsized":"2023-07-14T17:43:45.324438+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:15:31.705985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":44,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894178+0000","last_change":"2023-07-14T17:42:41.917282+0000","last_active":"2023-07-14T17:43:44.894178+0000","last_peered":"2023-07-14T17:43:44.894178+0000","last_clean":"2023-07-14T17:43:44.894178+0000","last_became_active":"2023-07-14T17:42:41.916994+0000","last_became_peered":"2023-07-14T17:42:41.916994+0000","last_unstale":"2023-07-14T17:43:44.894178+0000","last_undegraded":"2023-07-14T17:43:44.894178+0000","last_fullsized":"2023-07-14T17:43:44.894178+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:44:02.404111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897640+0000","last_change":"2023-07-14T17:42:42.548525+0000","last_active":"2023-07-14T17:43:44.897640+0000","last_peered":"2023-07-14T17:43:44.897640+0000","last_clean":"2023-07-14T17:43:44.897640+0000","last_became_active":"2023-07-14T17:42:42.548270+0000","last_became_peered":"2023-07-14T17:42:42.548270+0000","last_unstale":"2023-07-14T17:43:44.897640+0000","last_undegraded":"2023-07-14T17:43:44.897640+0000","last_fullsized":"2023-07-14T17:43:44.897640+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:14:31.331118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.898128+0000","last_change":"2023-07-14T17:42:46.546480+0000","last_active":"2023-07-14T17:43:44.898128+0000","last_peered":"2023-07-14T17:43:44.898128+0000","last_clean":"2023-07-14T17:43:44.898128+0000","last_became_active":"2023-07-14T17:42:46.546078+0000","last_became_peered":"2023-07-14T17:42:46.546078+0000","last_unstale":"2023-07-14T17:43:44.898128+0000","last_undegraded":"2023-07-14T17:43:44.898128+0000","last_fullsized":"2023-07-14T17:43:44.898128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:41:01.115708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894258+0000","last_change":"2023-07-14T17:42:41.914226+0000","last_active":"2023-07-14T17:43:44.894258+0000","last_peered":"2023-07-14T17:43:44.894258+0000","last_clean":"2023-07-14T17:43:44.894258+0000","last_became_active":"2023-07-14T17:42:41.913820+0000","last_became_peered":"2023-07-14T17:42:41.913820+0000","last_unstale":"2023-07-14T17:43:44.894258+0000","last_undegraded":"2023-07-14T17:43:44.894258+0000","last_fullsized":"2023-07-14T17:43:44.894258+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:21:15.638447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892019+0000","last_change":"2023-07-14T17:42:42.543549+0000","last_active":"2023-07-14T17:43:44.892019+0000","last_peered":"2023-07-14T17:43:44.892019+0000","last_clean":"2023-07-14T17:43:44.892019+0000","last_became_active":"2023-07-14T17:42:42.543245+0000","last_became_peered":"2023-07-14T17:42:42.543245+0000","last_unstale":"2023-07-14T17:43:44.892019+0000","last_undegraded":"2023-07-14T17:43:44.892019+0000","last_fullsized":"2023-07-14T17:43:44.892019+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:18:13.987338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897150+0000","last_change":"2023-07-14T17:42:46.577245+0000","last_active":"2023-07-14T17:43:44.897150+0000","last_peered":"2023-07-14T17:43:44.897150+0000","last_clean":"2023-07-14T17:43:44.897150+0000","last_became_active":"2023-07-14T17:42:46.576793+0000","last_became_peered":"2023-07-14T17:42:46.576793+0000","last_unstale":"2023-07-14T17:43:44.897150+0000","last_undegraded":"2023-07-14T17:43:44.897150+0000","last_fullsized":"2023-07-14T17:43:44.897150+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:22:16.800489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892975+0000","last_change":"2023-07-14T17:42:41.917199+0000","last_active":"2023-07-14T17:43:44.892975+0000","last_peered":"2023-07-14T17:43:44.892975+0000","last_clean":"2023-07-14T17:43:44.892975+0000","last_became_active":"2023-07-14T17:42:41.916924+0000","last_became_peered":"2023-07-14T17:42:41.916924+0000","last_unstale":"2023-07-14T17:43:44.892975+0000","last_undegraded":"2023-07-14T17:43:44.892975+0000","last_fullsized":"2023-07-14T17:43:44.892975+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:10:08.589818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896869+0000","last_change":"2023-07-14T17:42:42.540531+0000","last_active":"2023-07-14T17:43:44.896869+0000","last_peered":"2023-07-14T17:43:44.896869+0000","last_clean":"2023-07-14T17:43:44.896869+0000","last_became_active":"2023-07-14T17:42:42.540165+0000","last_became_peered":"2023-07-14T17:42:42.540165+0000","last_unstale":"2023-07-14T17:43:44.896869+0000","last_undegraded":"2023-07-14T17:43:44.896869+0000","last_fullsized":"2023-07-14T17:43:44.896869+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:30:36.053357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324213+0000","last_change":"2023-07-14T17:42:46.537158+0000","last_active":"2023-07-14T17:43:45.324213+0000","last_peered":"2023-07-14T17:43:45.324213+0000","last_clean":"2023-07-14T17:43:45.324213+0000","last_became_active":"2023-07-14T17:42:46.536816+0000","last_became_peered":"2023-07-14T17:42:46.536816+0000","last_unstale":"2023-07-14T17:43:45.324213+0000","last_undegraded":"2023-07-14T17:43:45.324213+0000","last_fullsized":"2023-07-14T17:43:45.324213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:00:28.047931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":84,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:54.870949+0000","last_change":"2023-07-14T17:42:41.913638+0000","last_active":"2023-07-14T17:43:54.870949+0000","last_peered":"2023-07-14T17:43:54.870949+0000","last_clean":"2023-07-14T17:43:54.870949+0000","last_became_active":"2023-07-14T17:42:41.913361+0000","last_became_peered":"2023-07-14T17:42:41.913361+0000","last_unstale":"2023-07-14T17:43:54.870949+0000","last_undegraded":"2023-07-14T17:43:54.870949+0000","last_fullsized":"2023-07-14T17:43:54.870949+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:46:04.106529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":53,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.901414+0000","last_change":"2023-07-14T17:43:44.901414+0000","last_active":"2023-07-14T17:43:44.901414+0000","last_peered":"2023-07-14T17:43:44.901414+0000","last_clean":"2023-07-14T17:43:44.901414+0000","last_became_active":"2023-07-14T17:43:44.901164+0000","last_became_peered":"2023-07-14T17:43:44.901164+0000","last_unstale":"2023-07-14T17:43:44.901414+0000","last_undegraded":"2023-07-14T17:43:44.901414+0000","last_fullsized":"2023-07-14T17:43:44.901414+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:15:25.526757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324220+0000","last_change":"2023-07-14T17:42:46.576999+0000","last_active":"2023-07-14T17:43:45.324220+0000","last_peered":"2023-07-14T17:43:45.324220+0000","last_clean":"2023-07-14T17:43:45.324220+0000","last_became_active":"2023-07-14T17:42:46.576652+0000","last_became_peered":"2023-07-14T17:42:46.576652+0000","last_unstale":"2023-07-14T17:43:45.324220+0000","last_undegraded":"2023-07-14T17:43:45.324220+0000","last_fullsized":"2023-07-14T17:43:45.324220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:14.124797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897667+0000","last_change":"2023-07-14T17:42:42.547053+0000","last_active":"2023-07-14T17:43:44.897667+0000","last_peered":"2023-07-14T17:43:44.897667+0000","last_clean":"2023-07-14T17:43:44.897667+0000","last_became_active":"2023-07-14T17:42:42.546735+0000","last_became_peered":"2023-07-14T17:42:42.546735+0000","last_unstale":"2023-07-14T17:43:44.897667+0000","last_undegraded":"2023-07-14T17:43:44.897667+0000","last_fullsized":"2023-07-14T17:43:44.897667+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:48:04.951463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":83,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:53.868986+0000","last_change":"2023-07-14T17:42:41.918692+0000","last_active":"2023-07-14T17:43:53.868986+0000","last_peered":"2023-07-14T17:43:53.868986+0000","last_clean":"2023-07-14T17:43:53.868986+0000","last_became_active":"2023-07-14T17:42:41.918360+0000","last_became_peered":"2023-07-14T17:42:41.918360+0000","last_unstale":"2023-07-14T17:43:53.868986+0000","last_undegraded":"2023-07-14T17:43:53.868986+0000","last_fullsized":"2023-07-14T17:43:53.868986+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:13:06.540310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":52,"num_read_kb":52,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894327+0000","last_change":"2023-07-14T17:42:46.543997+0000","last_active":"2023-07-14T17:43:44.894327+0000","last_peered":"2023-07-14T17:43:44.894327+0000","last_clean":"2023-07-14T17:43:44.894327+0000","last_became_active":"2023-07-14T17:42:46.543650+0000","last_became_peered":"2023-07-14T17:42:46.543650+0000","last_unstale":"2023-07-14T17:43:44.894327+0000","last_undegraded":"2023-07-14T17:43:44.894327+0000","last_fullsized":"2023-07-14T17:43:44.894327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:30:06.964324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895872+0000","last_change":"2023-07-14T17:42:42.548860+0000","last_active":"2023-07-14T17:43:44.895872+0000","last_peered":"2023-07-14T17:43:44.895872+0000","last_clean":"2023-07-14T17:43:44.895872+0000","last_became_active":"2023-07-14T17:42:42.548564+0000","last_became_peered":"2023-07-14T17:42:42.548564+0000","last_unstale":"2023-07-14T17:43:44.895872+0000","last_undegraded":"2023-07-14T17:43:44.895872+0000","last_fullsized":"2023-07-14T17:43:44.895872+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:51:19.885558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.813974+0000","last_change":"2023-07-14T17:42:41.912601+0000","last_active":"2023-07-14T17:43:45.813974+0000","last_peered":"2023-07-14T17:43:45.813974+0000","last_clean":"2023-07-14T17:43:45.813974+0000","last_became_active":"2023-07-14T17:42:41.912265+0000","last_became_peered":"2023-07-14T17:42:41.912265+0000","last_unstale":"2023-07-14T17:43:45.813974+0000","last_undegraded":"2023-07-14T17:43:45.813974+0000","last_fullsized":"2023-07-14T17:43:45.813974+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:20:08.059396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":347,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894007+0000","last_change":"2023-07-14T17:41:43.696093+0000","last_active":"2023-07-14T17:43:44.894007+0000","last_peered":"2023-07-14T17:43:44.894007+0000","last_clean":"2023-07-14T17:43:44.894007+0000","last_became_active":"2023-07-14T17:41:43.695511+0000","last_became_peered":"2023-07-14T17:41:43.695511+0000","last_unstale":"2023-07-14T17:43:44.894007+0000","last_undegraded":"2023-07-14T17:43:44.894007+0000","last_fullsized":"2023-07-14T17:43:44.894007+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:38:50.657865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:38:50.657865+0000","last_clean_scrub_stamp":"2023-07-14T17:38:50.657865+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:01:26.054999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.325941+0000","last_change":"2023-07-14T17:43:45.325941+0000","last_active":"2023-07-14T17:43:45.325941+0000","last_peered":"2023-07-14T17:43:45.325941+0000","last_clean":"2023-07-14T17:43:45.325941+0000","last_became_active":"2023-07-14T17:43:45.325519+0000","last_became_peered":"2023-07-14T17:43:45.325519+0000","last_unstale":"2023-07-14T17:43:45.325941+0000","last_undegraded":"2023-07-14T17:43:45.325941+0000","last_fullsized":"2023-07-14T17:43:45.325941+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:06:30.535779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,6,10],"acting":[5,11,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.323905+0000","last_change":"2023-07-14T17:42:42.543243+0000","last_active":"2023-07-14T17:43:45.323905+0000","last_peered":"2023-07-14T17:43:45.323905+0000","last_clean":"2023-07-14T17:43:45.323905+0000","last_became_active":"2023-07-14T17:42:42.542965+0000","last_became_peered":"2023-07-14T17:42:42.542965+0000","last_unstale":"2023-07-14T17:43:45.323905+0000","last_undegraded":"2023-07-14T17:43:45.323905+0000","last_fullsized":"2023-07-14T17:43:45.323905+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:11:31.220902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894165+0000","last_change":"2023-07-14T17:42:41.920806+0000","last_active":"2023-07-14T17:43:44.894165+0000","last_peered":"2023-07-14T17:43:44.894165+0000","last_clean":"2023-07-14T17:43:44.894165+0000","last_became_active":"2023-07-14T17:42:41.920528+0000","last_became_peered":"2023-07-14T17:42:41.920528+0000","last_unstale":"2023-07-14T17:43:44.894165+0000","last_undegraded":"2023-07-14T17:43:44.894165+0000","last_fullsized":"2023-07-14T17:43:44.894165+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:56:57.810566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.891910+0000","last_change":"2023-07-14T17:42:46.542026+0000","last_active":"2023-07-14T17:43:44.891910+0000","last_peered":"2023-07-14T17:43:44.891910+0000","last_clean":"2023-07-14T17:43:44.891910+0000","last_became_active":"2023-07-14T17:42:46.541672+0000","last_became_peered":"2023-07-14T17:42:46.541672+0000","last_unstale":"2023-07-14T17:43:44.891910+0000","last_undegraded":"2023-07-14T17:43:44.891910+0000","last_fullsized":"2023-07-14T17:43:44.891910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:31:32.137890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896620+0000","last_change":"2023-07-14T17:42:41.915798+0000","last_active":"2023-07-14T17:43:44.896620+0000","last_peered":"2023-07-14T17:43:44.896620+0000","last_clean":"2023-07-14T17:43:44.896620+0000","last_became_active":"2023-07-14T17:42:41.915492+0000","last_became_peered":"2023-07-14T17:42:41.915492+0000","last_unstale":"2023-07-14T17:43:44.896620+0000","last_undegraded":"2023-07-14T17:43:44.896620+0000","last_fullsized":"2023-07-14T17:43:44.896620+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:08:22.653303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324401+0000","last_change":"2023-07-14T17:42:42.533500+0000","last_active":"2023-07-14T17:43:45.324401+0000","last_peered":"2023-07-14T17:43:45.324401+0000","last_clean":"2023-07-14T17:43:45.324401+0000","last_became_active":"2023-07-14T17:42:42.533232+0000","last_became_peered":"2023-07-14T17:42:42.533232+0000","last_unstale":"2023-07-14T17:43:45.324401+0000","last_undegraded":"2023-07-14T17:43:45.324401+0000","last_fullsized":"2023-07-14T17:43:45.324401+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:42:49.105557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895003+0000","last_change":"2023-07-14T17:42:46.577131+0000","last_active":"2023-07-14T17:43:44.895003+0000","last_peered":"2023-07-14T17:43:44.895003+0000","last_clean":"2023-07-14T17:43:44.895003+0000","last_became_active":"2023-07-14T17:42:46.576695+0000","last_became_peered":"2023-07-14T17:42:46.576695+0000","last_unstale":"2023-07-14T17:43:44.895003+0000","last_undegraded":"2023-07-14T17:43:44.895003+0000","last_fullsized":"2023-07-14T17:43:44.895003+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:46:57.153931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"86'9","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.893473+0000","last_change":"2023-07-14T17:42:41.919021+0000","last_active":"2023-07-14T17:43:44.893473+0000","last_peered":"2023-07-14T17:43:44.893473+0000","last_clean":"2023-07-14T17:43:44.893473+0000","last_became_active":"2023-07-14T17:42:41.918714+0000","last_became_peered":"2023-07-14T17:42:41.918714+0000","last_unstale":"2023-07-14T17:43:44.893473+0000","last_undegraded":"2023-07-14T17:43:44.893473+0000","last_fullsized":"2023-07-14T17:43:44.893473+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:11:20.666027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897256+0000","last_change":"2023-07-14T17:42:42.534661+0000","last_active":"2023-07-14T17:43:44.897256+0000","last_peered":"2023-07-14T17:43:44.897256+0000","last_clean":"2023-07-14T17:43:44.897256+0000","last_became_active":"2023-07-14T17:42:42.534357+0000","last_became_peered":"2023-07-14T17:42:42.534357+0000","last_unstale":"2023-07-14T17:43:44.897256+0000","last_undegraded":"2023-07-14T17:43:44.897256+0000","last_fullsized":"2023-07-14T17:43:44.897256+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:13:04.290337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894357+0000","last_change":"2023-07-14T17:42:46.538648+0000","last_active":"2023-07-14T17:43:44.894357+0000","last_peered":"2023-07-14T17:43:44.894357+0000","last_clean":"2023-07-14T17:43:44.894357+0000","last_became_active":"2023-07-14T17:42:46.538225+0000","last_became_peered":"2023-07-14T17:42:46.538225+0000","last_unstale":"2023-07-14T17:43:44.894357+0000","last_undegraded":"2023-07-14T17:43:44.894357+0000","last_fullsized":"2023-07-14T17:43:44.894357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:21:00.195671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.814243+0000","last_change":"2023-07-14T17:42:41.915406+0000","last_active":"2023-07-14T17:43:45.814243+0000","last_peered":"2023-07-14T17:43:45.814243+0000","last_clean":"2023-07-14T17:43:45.814243+0000","last_became_active":"2023-07-14T17:42:41.914967+0000","last_became_peered":"2023-07-14T17:42:41.914967+0000","last_unstale":"2023-07-14T17:43:45.814243+0000","last_undegraded":"2023-07-14T17:43:45.814243+0000","last_fullsized":"2023-07-14T17:43:45.814243+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:38:20.282477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.325475+0000","last_change":"2023-07-14T17:43:45.325475+0000","last_active":"2023-07-14T17:43:45.325475+0000","last_peered":"2023-07-14T17:43:45.325475+0000","last_clean":"2023-07-14T17:43:45.325475+0000","last_became_active":"2023-07-14T17:43:45.325074+0000","last_became_peered":"2023-07-14T17:43:45.325074+0000","last_unstale":"2023-07-14T17:43:45.325475+0000","last_undegraded":"2023-07-14T17:43:45.325475+0000","last_fullsized":"2023-07-14T17:43:45.325475+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:45:33.551379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.814371+0000","last_change":"2023-07-14T17:42:46.577224+0000","last_active":"2023-07-14T17:43:45.814371+0000","last_peered":"2023-07-14T17:43:45.814371+0000","last_clean":"2023-07-14T17:43:45.814371+0000","last_became_active":"2023-07-14T17:42:46.576781+0000","last_became_peered":"2023-07-14T17:42:46.576781+0000","last_unstale":"2023-07-14T17:43:45.814371+0000","last_undegraded":"2023-07-14T17:43:45.814371+0000","last_fullsized":"2023-07-14T17:43:45.814371+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:03:30.154940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896001+0000","last_change":"2023-07-14T17:42:41.916406+0000","last_active":"2023-07-14T17:43:44.896001+0000","last_peered":"2023-07-14T17:43:44.896001+0000","last_clean":"2023-07-14T17:43:44.896001+0000","last_became_active":"2023-07-14T17:42:41.916115+0000","last_became_peered":"2023-07-14T17:42:41.916115+0000","last_unstale":"2023-07-14T17:43:44.896001+0000","last_undegraded":"2023-07-14T17:43:44.896001+0000","last_fullsized":"2023-07-14T17:43:44.896001+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:46:09.237959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894118+0000","last_change":"2023-07-14T17:42:42.529969+0000","last_active":"2023-07-14T17:43:44.894118+0000","last_peered":"2023-07-14T17:43:44.894118+0000","last_clean":"2023-07-14T17:43:44.894118+0000","last_became_active":"2023-07-14T17:42:42.529659+0000","last_became_peered":"2023-07-14T17:42:42.529659+0000","last_unstale":"2023-07-14T17:43:44.894118+0000","last_undegraded":"2023-07-14T17:43:44.894118+0000","last_fullsized":"2023-07-14T17:43:44.894118+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:36:09.137465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.893324+0000","last_change":"2023-07-14T17:42:46.546303+0000","last_active":"2023-07-14T17:43:44.893324+0000","last_peered":"2023-07-14T17:43:44.893324+0000","last_clean":"2023-07-14T17:43:44.893324+0000","last_became_active":"2023-07-14T17:42:46.545947+0000","last_became_peered":"2023-07-14T17:42:46.545947+0000","last_unstale":"2023-07-14T17:43:44.893324+0000","last_undegraded":"2023-07-14T17:43:44.893324+0000","last_fullsized":"2023-07-14T17:43:44.893324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:21:36.193257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895642+0000","last_change":"2023-07-14T17:42:41.916373+0000","last_active":"2023-07-14T17:43:44.895642+0000","last_peered":"2023-07-14T17:43:44.895642+0000","last_clean":"2023-07-14T17:43:44.895642+0000","last_became_active":"2023-07-14T17:42:41.915966+0000","last_became_peered":"2023-07-14T17:42:41.915966+0000","last_unstale":"2023-07-14T17:43:44.895642+0000","last_undegraded":"2023-07-14T17:43:44.895642+0000","last_fullsized":"2023-07-14T17:43:44.895642+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:32:53.280585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.891773+0000","last_change":"2023-07-14T17:42:42.532040+0000","last_active":"2023-07-14T17:43:44.891773+0000","last_peered":"2023-07-14T17:43:44.891773+0000","last_clean":"2023-07-14T17:43:44.891773+0000","last_became_active":"2023-07-14T17:42:42.531690+0000","last_became_peered":"2023-07-14T17:42:42.531690+0000","last_unstale":"2023-07-14T17:43:44.891773+0000","last_undegraded":"2023-07-14T17:43:44.891773+0000","last_fullsized":"2023-07-14T17:43:44.891773+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:19:44.676745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.814020+0000","last_change":"2023-07-14T17:42:46.550166+0000","last_active":"2023-07-14T17:43:45.814020+0000","last_peered":"2023-07-14T17:43:45.814020+0000","last_clean":"2023-07-14T17:43:45.814020+0000","last_became_active":"2023-07-14T17:42:46.549700+0000","last_became_peered":"2023-07-14T17:42:46.549700+0000","last_unstale":"2023-07-14T17:43:45.814020+0000","last_undegraded":"2023-07-14T17:43:45.814020+0000","last_fullsized":"2023-07-14T17:43:45.814020+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:41:53.732450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.911523+0000","last_change":"2023-07-14T17:43:44.911267+0000","last_active":"2023-07-14T17:43:44.911523+0000","last_peered":"2023-07-14T17:43:44.911523+0000","last_clean":"2023-07-14T17:43:44.911523+0000","last_became_active":"2023-07-14T17:43:44.899552+0000","last_became_peered":"2023-07-14T17:43:44.899552+0000","last_unstale":"2023-07-14T17:43:44.911523+0000","last_undegraded":"2023-07-14T17:43:44.911523+0000","last_fullsized":"2023-07-14T17:43:44.911523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:30:32.991380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897748+0000","last_change":"2023-07-14T17:42:42.531789+0000","last_active":"2023-07-14T17:43:44.897748+0000","last_peered":"2023-07-14T17:43:44.897748+0000","last_clean":"2023-07-14T17:43:44.897748+0000","last_became_active":"2023-07-14T17:42:42.531460+0000","last_became_peered":"2023-07-14T17:42:42.531460+0000","last_unstale":"2023-07-14T17:43:44.897748+0000","last_undegraded":"2023-07-14T17:43:44.897748+0000","last_fullsized":"2023-07-14T17:43:44.897748+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:46.112701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.814241+0000","last_change":"2023-07-14T17:42:46.577034+0000","last_active":"2023-07-14T17:43:45.814241+0000","last_peered":"2023-07-14T17:43:45.814241+0000","last_clean":"2023-07-14T17:43:45.814241+0000","last_became_active":"2023-07-14T17:42:46.576636+0000","last_became_peered":"2023-07-14T17:42:46.576636+0000","last_unstale":"2023-07-14T17:43:45.814241+0000","last_undegraded":"2023-07-14T17:43:45.814241+0000","last_fullsized":"2023-07-14T17:43:45.814241+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:40:08.514523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894683+0000","last_change":"2023-07-14T17:42:41.919099+0000","last_active":"2023-07-14T17:43:44.894683+0000","last_peered":"2023-07-14T17:43:44.894683+0000","last_clean":"2023-07-14T17:43:44.894683+0000","last_became_active":"2023-07-14T17:42:41.918306+0000","last_became_peered":"2023-07-14T17:42:41.918306+0000","last_unstale":"2023-07-14T17:43:44.894683+0000","last_undegraded":"2023-07-14T17:43:44.894683+0000","last_fullsized":"2023-07-14T17:43:44.894683+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:38:47.955835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894358+0000","last_change":"2023-07-14T17:42:42.552969+0000","last_active":"2023-07-14T17:43:44.894358+0000","last_peered":"2023-07-14T17:43:44.894358+0000","last_clean":"2023-07-14T17:43:44.894358+0000","last_became_active":"2023-07-14T17:42:42.552647+0000","last_became_peered":"2023-07-14T17:42:42.552647+0000","last_unstale":"2023-07-14T17:43:44.894358+0000","last_undegraded":"2023-07-14T17:43:44.894358+0000","last_fullsized":"2023-07-14T17:43:44.894358+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:41:32.606687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.896695+0000","last_change":"2023-07-14T17:42:46.538335+0000","last_active":"2023-07-14T17:43:44.896695+0000","last_peered":"2023-07-14T17:43:44.896695+0000","last_clean":"2023-07-14T17:43:44.896695+0000","last_became_active":"2023-07-14T17:42:46.537929+0000","last_became_peered":"2023-07-14T17:42:46.537929+0000","last_unstale":"2023-07-14T17:43:44.896695+0000","last_undegraded":"2023-07-14T17:43:44.896695+0000","last_fullsized":"2023-07-14T17:43:44.896695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:03:03.278360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:53.887906+0000","last_change":"2023-07-14T17:42:41.916323+0000","last_active":"2023-07-14T17:43:53.887906+0000","last_peered":"2023-07-14T17:43:53.887906+0000","last_clean":"2023-07-14T17:43:53.887906+0000","last_became_active":"2023-07-14T17:42:41.916019+0000","last_became_peered":"2023-07-14T17:42:41.916019+0000","last_unstale":"2023-07-14T17:43:53.887906+0000","last_undegraded":"2023-07-14T17:43:53.887906+0000","last_fullsized":"2023-07-14T17:43:53.887906+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:29:56.282750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1122,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":53,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894518+0000","last_change":"2023-07-14T17:42:42.549843+0000","last_active":"2023-07-14T17:43:44.894518+0000","last_peered":"2023-07-14T17:43:44.894518+0000","last_clean":"2023-07-14T17:43:44.894518+0000","last_became_active":"2023-07-14T17:42:42.549347+0000","last_became_peered":"2023-07-14T17:42:42.549347+0000","last_unstale":"2023-07-14T17:43:44.894518+0000","last_undegraded":"2023-07-14T17:43:44.894518+0000","last_fullsized":"2023-07-14T17:43:44.894518+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:32:23.560638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.898526+0000","last_change":"2023-07-14T17:42:46.545262+0000","last_active":"2023-07-14T17:43:44.898526+0000","last_peered":"2023-07-14T17:43:44.898526+0000","last_clean":"2023-07-14T17:43:44.898526+0000","last_became_active":"2023-07-14T17:42:46.544891+0000","last_became_peered":"2023-07-14T17:42:46.544891+0000","last_unstale":"2023-07-14T17:43:44.898526+0000","last_undegraded":"2023-07-14T17:43:44.898526+0000","last_fullsized":"2023-07-14T17:43:44.898526+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:58:48.061797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897783+0000","last_change":"2023-07-14T17:42:41.912855+0000","last_active":"2023-07-14T17:43:44.897783+0000","last_peered":"2023-07-14T17:43:44.897783+0000","last_clean":"2023-07-14T17:43:44.897783+0000","last_became_active":"2023-07-14T17:42:41.912510+0000","last_became_peered":"2023-07-14T17:42:41.912510+0000","last_unstale":"2023-07-14T17:43:44.897783+0000","last_undegraded":"2023-07-14T17:43:44.897783+0000","last_fullsized":"2023-07-14T17:43:44.897783+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:29:49.003962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894208+0000","last_change":"2023-07-14T17:42:42.545344+0000","last_active":"2023-07-14T17:43:44.894208+0000","last_peered":"2023-07-14T17:43:44.894208+0000","last_clean":"2023-07-14T17:43:44.894208+0000","last_became_active":"2023-07-14T17:42:42.545056+0000","last_became_peered":"2023-07-14T17:42:42.545056+0000","last_unstale":"2023-07-14T17:43:44.894208+0000","last_undegraded":"2023-07-14T17:43:44.894208+0000","last_fullsized":"2023-07-14T17:43:44.894208+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:31:59.187718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897362+0000","last_change":"2023-07-14T17:42:46.541279+0000","last_active":"2023-07-14T17:43:44.897362+0000","last_peered":"2023-07-14T17:43:44.897362+0000","last_clean":"2023-07-14T17:43:44.897362+0000","last_became_active":"2023-07-14T17:42:46.540835+0000","last_became_peered":"2023-07-14T17:42:46.540835+0000","last_unstale":"2023-07-14T17:43:44.897362+0000","last_undegraded":"2023-07-14T17:43:44.897362+0000","last_fullsized":"2023-07-14T17:43:44.897362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:45.521346+0000","last_clean_scrub_stamp":"2023-07-14T17:42:45.521346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:41:42.325967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324578+0000","last_change":"2023-07-14T17:42:41.919257+0000","last_active":"2023-07-14T17:43:45.324578+0000","last_peered":"2023-07-14T17:43:45.324578+0000","last_clean":"2023-07-14T17:43:45.324578+0000","last_became_active":"2023-07-14T17:42:41.918968+0000","last_became_peered":"2023-07-14T17:42:41.918968+0000","last_unstale":"2023-07-14T17:43:45.324578+0000","last_undegraded":"2023-07-14T17:43:45.324578+0000","last_fullsized":"2023-07-14T17:43:45.324578+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:40.890180+0000","last_clean_scrub_stamp":"2023-07-14T17:42:40.890180+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:06:24.227953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894456+0000","last_change":"2023-07-14T17:42:42.543993+0000","last_active":"2023-07-14T17:43:44.894456+0000","last_peered":"2023-07-14T17:43:44.894456+0000","last_clean":"2023-07-14T17:43:44.894456+0000","last_became_active":"2023-07-14T17:42:42.543685+0000","last_became_peered":"2023-07-14T17:42:42.543685+0000","last_unstale":"2023-07-14T17:43:44.894456+0000","last_undegraded":"2023-07-14T17:43:44.894456+0000","last_fullsized":"2023-07-14T17:43:44.894456+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:45:26.338845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.891847+0000","last_change":"2023-07-14T17:42:42.538553+0000","last_active":"2023-07-14T17:43:44.891847+0000","last_peered":"2023-07-14T17:43:44.891847+0000","last_clean":"2023-07-14T17:43:44.891847+0000","last_became_active":"2023-07-14T17:42:42.538251+0000","last_became_peered":"2023-07-14T17:42:42.538251+0000","last_unstale":"2023-07-14T17:43:44.891847+0000","last_undegraded":"2023-07-14T17:43:44.891847+0000","last_fullsized":"2023-07-14T17:43:44.891847+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:44:50.672379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894621+0000","last_change":"2023-07-14T17:42:42.543819+0000","last_active":"2023-07-14T17:43:44.894621+0000","last_peered":"2023-07-14T17:43:44.894621+0000","last_clean":"2023-07-14T17:43:44.894621+0000","last_became_active":"2023-07-14T17:42:42.543311+0000","last_became_peered":"2023-07-14T17:42:42.543311+0000","last_unstale":"2023-07-14T17:43:44.894621+0000","last_undegraded":"2023-07-14T17:43:44.894621+0000","last_fullsized":"2023-07-14T17:43:44.894621+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:03:11.132113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.894009+0000","last_change":"2023-07-14T17:42:42.537525+0000","last_active":"2023-07-14T17:43:44.894009+0000","last_peered":"2023-07-14T17:43:44.894009+0000","last_clean":"2023-07-14T17:43:44.894009+0000","last_became_active":"2023-07-14T17:42:42.537242+0000","last_became_peered":"2023-07-14T17:42:42.537242+0000","last_unstale":"2023-07-14T17:43:44.894009+0000","last_undegraded":"2023-07-14T17:43:44.894009+0000","last_fullsized":"2023-07-14T17:43:44.894009+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:40:58.077332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.898807+0000","last_change":"2023-07-14T17:42:42.547670+0000","last_active":"2023-07-14T17:43:44.898807+0000","last_peered":"2023-07-14T17:43:44.898807+0000","last_clean":"2023-07-14T17:43:44.898807+0000","last_became_active":"2023-07-14T17:42:42.547349+0000","last_became_peered":"2023-07-14T17:42:42.547349+0000","last_unstale":"2023-07-14T17:43:44.898807+0000","last_undegraded":"2023-07-14T17:43:44.898807+0000","last_fullsized":"2023-07-14T17:43:44.898807+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:49:40.819754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324379+0000","last_change":"2023-07-14T17:42:42.542741+0000","last_active":"2023-07-14T17:43:45.324379+0000","last_peered":"2023-07-14T17:43:45.324379+0000","last_clean":"2023-07-14T17:43:45.324379+0000","last_became_active":"2023-07-14T17:42:42.541972+0000","last_became_peered":"2023-07-14T17:42:42.541972+0000","last_unstale":"2023-07-14T17:43:45.324379+0000","last_undegraded":"2023-07-14T17:43:45.324379+0000","last_fullsized":"2023-07-14T17:43:45.324379+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:27.843013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.892205+0000","last_change":"2023-07-14T17:42:42.535043+0000","last_active":"2023-07-14T17:43:44.892205+0000","last_peered":"2023-07-14T17:43:44.892205+0000","last_clean":"2023-07-14T17:43:44.892205+0000","last_became_active":"2023-07-14T17:42:42.534281+0000","last_became_peered":"2023-07-14T17:42:42.534281+0000","last_unstale":"2023-07-14T17:43:44.892205+0000","last_undegraded":"2023-07-14T17:43:44.892205+0000","last_fullsized":"2023-07-14T17:43:44.892205+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:27:45.984466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895460+0000","last_change":"2023-07-14T17:42:42.540866+0000","last_active":"2023-07-14T17:43:44.895460+0000","last_peered":"2023-07-14T17:43:44.895460+0000","last_clean":"2023-07-14T17:43:44.895460+0000","last_became_active":"2023-07-14T17:42:42.540568+0000","last_became_peered":"2023-07-14T17:42:42.540568+0000","last_unstale":"2023-07-14T17:43:44.895460+0000","last_undegraded":"2023-07-14T17:43:44.895460+0000","last_fullsized":"2023-07-14T17:43:44.895460+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:12:01.698385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897437+0000","last_change":"2023-07-14T17:42:42.541646+0000","last_active":"2023-07-14T17:43:44.897437+0000","last_peered":"2023-07-14T17:43:44.897437+0000","last_clean":"2023-07-14T17:43:44.897437+0000","last_became_active":"2023-07-14T17:42:42.541324+0000","last_became_peered":"2023-07-14T17:42:42.541324+0000","last_unstale":"2023-07-14T17:43:44.897437+0000","last_undegraded":"2023-07-14T17:43:44.897437+0000","last_fullsized":"2023-07-14T17:43:44.897437+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:37:25.561305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897528+0000","last_change":"2023-07-14T17:42:42.534650+0000","last_active":"2023-07-14T17:43:44.897528+0000","last_peered":"2023-07-14T17:43:44.897528+0000","last_clean":"2023-07-14T17:43:44.897528+0000","last_became_active":"2023-07-14T17:42:42.534351+0000","last_became_peered":"2023-07-14T17:42:42.534351+0000","last_unstale":"2023-07-14T17:43:44.897528+0000","last_undegraded":"2023-07-14T17:43:44.897528+0000","last_fullsized":"2023-07-14T17:43:44.897528+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:13:52.397350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.895517+0000","last_change":"2023-07-14T17:42:42.535815+0000","last_active":"2023-07-14T17:43:44.895517+0000","last_peered":"2023-07-14T17:43:44.895517+0000","last_clean":"2023-07-14T17:43:44.895517+0000","last_became_active":"2023-07-14T17:42:42.535510+0000","last_became_peered":"2023-07-14T17:42:42.535510+0000","last_unstale":"2023-07-14T17:43:44.895517+0000","last_undegraded":"2023-07-14T17:43:44.895517+0000","last_fullsized":"2023-07-14T17:43:44.895517+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T17:52:40.052652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.897009+0000","last_change":"2023-07-14T17:42:42.551758+0000","last_active":"2023-07-14T17:43:44.897009+0000","last_peered":"2023-07-14T17:43:44.897009+0000","last_clean":"2023-07-14T17:43:44.897009+0000","last_became_active":"2023-07-14T17:42:42.551444+0000","last_became_peered":"2023-07-14T17:42:42.551444+0000","last_unstale":"2023-07-14T17:43:44.897009+0000","last_undegraded":"2023-07-14T17:43:44.897009+0000","last_fullsized":"2023-07-14T17:43:44.897009+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:26:22.278980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:45.324115+0000","last_change":"2023-07-14T17:42:42.546355+0000","last_active":"2023-07-14T17:43:45.324115+0000","last_peered":"2023-07-14T17:43:45.324115+0000","last_clean":"2023-07-14T17:43:45.324115+0000","last_became_active":"2023-07-14T17:42:42.546079+0000","last_became_peered":"2023-07-14T17:42:42.546079+0000","last_unstale":"2023-07-14T17:43:45.324115+0000","last_undegraded":"2023-07-14T17:43:45.324115+0000","last_fullsized":"2023-07-14T17:43:45.324115+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:57:16.604025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-07-14T17:43:44.901014+0000","last_change":"2023-07-14T17:43:44.901014+0000","last_active":"2023-07-14T17:43:44.901014+0000","last_peered":"2023-07-14T17:43:44.901014+0000","last_clean":"2023-07-14T17:43:44.901014+0000","last_became_active":"2023-07-14T17:43:44.900632+0000","last_became_peered":"2023-07-14T17:43:44.900632+0000","last_unstale":"2023-07-14T17:43:44.901014+0000","last_undegraded":"2023-07-14T17:43:44.901014+0000","last_fullsized":"2023-07-14T17:43:44.901014+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T17:42:41.897055+0000","last_clean_scrub_stamp":"2023-07-14T17:42:41.897055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:23:03.050997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,8],"acting":[11,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65561,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":543,"num_read_kb":342,"num_write":234,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":196683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":206,"ondisk_log_size":206,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":77,"seq":330712481819,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":486248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.624}]},{"osd":1,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.48199999999999998}]},{"osd":2,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.46000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68700000000000006}]},{"osd":4,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"last":0.56000000000000005}]},{"osd":5,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.51700000000000002}]},{"osd":6,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.77500000000000002}]},{"osd":7,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.64300000000000002}]},{"osd":8,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.748}]},{"osd":9,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":1.4770000000000001},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.83799999999999997}]},{"osd":10,"last update":"Fri Jul 14 17:43:03 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.85499999999999998}]}]},{"osd":10,"up_from":69,"seq":296352743454,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28672,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716480,"statfs":{"total":95995035648,"available":95965675520,"internally_reserved":0,"allocated":1687552,"data_stored":1132005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.79800000000000004}]},{"osd":1,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.27500000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.77600000000000002,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.48299999999999998,"15min":0.47899999999999998},"min":{"1min":0.33400000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.54900000000000004}]},{"osd":2,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.59499999999999997,"15min":0.58499999999999996},"min":{"1min":0.51800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.83199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.629,"15min":0.627},"min":{"1min":0.48299999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.77300000000000002}]},{"osd":3,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.627,"15min":0.64300000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69699999999999995,"5min":0.876,"15min":0.876},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84899999999999998,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.91500000000000004}]},{"osd":4,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.61799999999999999,"15min":0.62},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.77500000000000002,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.54200000000000004,"15min":0.54000000000000004},"min":{"1min":0.41499999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.59999999999999998}]},{"osd":5,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.53200000000000003,"5min":0.434,"15min":0.434},"max":{"1min":0.83399999999999996,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.73799999999999999,"15min":0.746},"min":{"1min":0.53800000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.82899999999999996,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.89100000000000001}]},{"osd":6,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.72499999999999998,"15min":0.74099999999999999},"min":{"1min":0.41399999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78800000000000003,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.47299999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.86699999999999999,"5min":1.125,"15min":1.125},"last":0.98799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":8,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80000000000000004,"15min":0.79900000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.91000000000000003,"5min":1.204,"15min":1.204},"last":1.012},{"interface":"front","average":{"1min":0.75,"5min":0.76900000000000002,"15min":0.77200000000000002},"min":{"1min":0.60299999999999998,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88900000000000001,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.876}]},{"osd":9,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.82699999999999996,"15min":0.81799999999999995},"min":{"1min":0.65500000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.76200000000000001,"15min":0.77100000000000002},"min":{"1min":0.51300000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.92300000000000004,"5min":1.079,"15min":1.079},"last":0.93500000000000005}]},{"osd":11,"last update":"Fri Jul 14 17:43:05 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":1.0349999999999999}]}]},{"osd":9,"up_from":64,"seq":274877906978,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":490322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.41099999999999998,"15min":0.42399999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.497,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.45400000000000001,"15min":0.47199999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.55800000000000005,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.34799999999999998}]},{"osd":1,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.59099999999999997,"15min":0.60599999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.69899999999999995,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.60199999999999998,"15min":0.61599999999999999},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.70999999999999996,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.436}]},{"osd":2,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79900000000000004,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.64500000000000002,"15min":0.65700000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.73199999999999998,"5min":1.119,"15min":1.119},"last":0.64300000000000002}]},{"osd":3,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.48999999999999999,"15min":0.50600000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.52700000000000002,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.49199999999999999,"15min":0.505},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64400000000000002,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.316}]},{"osd":4,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.628,"15min":0.64300000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.70699999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.63600000000000001,"15min":0.64800000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.68300000000000005,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.45500000000000002}]},{"osd":5,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.64600000000000002,"15min":0.65800000000000003},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.78400000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.65800000000000003,"15min":0.67100000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79600000000000004,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.58199999999999996}]},{"osd":6,"last update":"Fri Jul 14 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.372},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.52500000000000002}]},{"osd":7,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.70599999999999996,"15min":0.72799999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71099999999999997,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.71699999999999997,"15min":0.73799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.79400000000000004,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.53800000000000003}]},{"osd":8,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.749,"15min":0.76800000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.82699999999999996,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.74199999999999999,"15min":0.75800000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.85899999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.63100000000000001}]},{"osd":10,"last update":"Fri Jul 14 17:43:40 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.73399999999999999,"15min":0.755},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.74399999999999999,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.73199999999999998,"15min":0.748},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.73999999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.48299999999999998}]},{"osd":11,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.65500000000000003}]}]},{"osd":8,"up_from":57,"seq":244813135911,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1015808,"data_stored":486604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.54300000000000004,"15min":0.54500000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.53400000000000003,"15min":0.53300000000000003},"min":{"1min":0.35499999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.55800000000000005}]},{"osd":1,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.372,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.83899999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.40999999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.70799999999999996}]},{"osd":2,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.45800000000000002,"15min":0.46100000000000002},"min":{"1min":0.27500000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60599999999999998,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.47199999999999998,"15min":0.47999999999999998},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.56499999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.39000000000000001}]},{"osd":3,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.36299999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.68200000000000005}]},{"osd":4,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85099999999999998,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64300000000000002,"15min":0.64200000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.89700000000000002,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.60799999999999998}]},{"osd":5,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57399999999999995,"15min":0.58099999999999996},"min":{"1min":0.34300000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.72499999999999998,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56999999999999995,"15min":0.57199999999999995},"min":{"1min":0.312,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.75,"5min":1.018,"15min":1.018},"last":0.45100000000000001}]},{"osd":6,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.66600000000000004,"15min":0.66800000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68500000000000005,"15min":0.69299999999999995},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.84799999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.57499999999999996}]},{"osd":7,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.51000000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92400000000000004,"5min":1.008,"15min":1.008},"last":0.746},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71099999999999997,"15min":0.70999999999999996},"min":{"1min":0.48899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.88200000000000001,"5min":1.258,"15min":1.258},"last":0.81100000000000005}]},{"osd":9,"last update":"Fri Jul 14 17:43:19 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.68500000000000005,"15min":0.67000000000000004},"min":{"1min":0.441,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.68600000000000005,"15min":0.67200000000000004},"min":{"1min":0.46899999999999997,"5min":0.44,"15min":0.44},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.83899999999999997}]},{"osd":10,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.69699999999999995,"15min":0.68100000000000005},"min":{"1min":0.57199999999999995,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.94399999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.70699999999999996,"15min":0.68500000000000005},"min":{"1min":0.61099999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.91000000000000003}]},{"osd":11,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.75800000000000001}]}]},{"osd":7,"up_from":49,"seq":210453397547,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1613824,"data_stored":1083346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.503},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.59499999999999997}]},{"osd":1,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.42199999999999999,"15min":0.42499999999999999},"min":{"1min":0.29999999999999999,"5min":0.25,"15min":0.25},"max":{"1min":0.70299999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.46600000000000003,"15min":0.47299999999999998},"min":{"1min":0.32400000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69499999999999995,"5min":0.746,"15min":0.746},"last":0.58499999999999996}]},{"osd":2,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.64300000000000002,"15min":0.60799999999999998},"min":{"1min":0.443,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82799999999999996,"5min":5.1050000000000004,"15min":5.1050000000000004},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.59799999999999998,"15min":0.58599999999999997},"min":{"1min":0.47099999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.83699999999999997,"5min":1.048,"15min":1.048},"last":0.63100000000000001}]},{"osd":3,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59299999999999997,"15min":0.59499999999999997},"min":{"1min":0.41799999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.86499999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.60899999999999999},"min":{"1min":0.432,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80700000000000005,"5min":0.995,"15min":0.995},"last":0.83699999999999997}]},{"osd":4,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.505,"15min":0.503},"min":{"1min":0.36799999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.76300000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.752},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52000000000000002,"15min":0.53200000000000003},"min":{"1min":0.307,"5min":0.248,"15min":0.248},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.504}]},{"osd":5,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.64800000000000002,"15min":0.63600000000000001},"min":{"1min":0.499,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98099999999999998,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.65600000000000003,"15min":0.64800000000000002},"min":{"1min":0.51600000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.96499999999999997,"5min":1.02,"15min":1.02},"last":0.55700000000000005}]},{"osd":6,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.64200000000000002,"15min":0.65000000000000002},"min":{"1min":0.45200000000000001,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.878,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.87},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.67600000000000005,"15min":0.68700000000000006},"min":{"1min":0.46700000000000003,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.94899999999999995,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.88200000000000001}]},{"osd":8,"last update":"Fri Jul 14 17:43:00 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.45700000000000002,"5min":0.441,"15min":0.441},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.71299999999999997,"15min":0.72099999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0620000000000001,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.82499999999999996}]},{"osd":9,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.69399999999999995,"15min":0.70199999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.01,"5min":1.196,"15min":1.196},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.91200000000000003,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.95799999999999996}]},{"osd":10,"last update":"Fri Jul 14 17:43:45 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.65800000000000003,"15min":0.63700000000000001},"min":{"1min":0.53400000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.67700000000000005,"15min":0.66100000000000003},"min":{"1min":0.505,"5min":0.441,"15min":0.441},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.72599999999999998}]},{"osd":11,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.90000000000000002}]}]},{"osd":6,"up_from":44,"seq":188978561071,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":995328,"data_stored":481682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.36799999999999999,"15min":0.35499999999999998},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.40300000000000002,"15min":0.39100000000000001},"min":{"1min":0.28199999999999997,"5min":0.255,"15min":0.255},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.29399999999999998}]},{"osd":1,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.57999999999999996,"15min":0.56799999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.76000000000000001,"5min":0.87,"15min":0.87},"last":0.5},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.56399999999999995,"15min":0.56100000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.65000000000000002,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.39200000000000002}]},{"osd":2,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.59399999999999997,"15min":0.58999999999999997},"min":{"1min":0.44500000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.55300000000000005}]},{"osd":3,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.44400000000000001,"15min":0.44700000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68500000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.308},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.47199999999999998,"15min":0.45400000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.78900000000000003,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.45100000000000001}]},{"osd":4,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85299999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.60799999999999998,"15min":0.60599999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.81799999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.62}]},{"osd":5,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.60099999999999998,"15min":0.59299999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.68600000000000005,"15min":0.66300000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":3.8679999999999999,"5min":3.8679999999999999,"15min":3.8679999999999999},"last":0.70799999999999996}]},{"osd":7,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.754,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.66600000000000004,"15min":0.68000000000000005},"min":{"1min":0.47999999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.73499999999999999,"5min":1.012,"15min":1.012},"last":0.52000000000000002}]},{"osd":8,"last update":"Fri Jul 14 17:43:04 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.73599999999999999,"15min":0.755},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.98999999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.69499999999999995,"15min":0.71099999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92500000000000004,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.60699999999999998}]},{"osd":9,"last update":"Fri Jul 14 17:43:24 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.68100000000000005,"15min":0.70099999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.91500000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.65700000000000003,"15min":0.67700000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.90200000000000002,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.47799999999999998}]},{"osd":10,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.72799999999999998,"15min":0.746},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.81999999999999995,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.755,"15min":0.76900000000000002},"min":{"1min":0.52500000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.83799999999999997,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.63700000000000001}]},{"osd":11,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.67200000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":541263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.503,"15min":0.46700000000000003},"min":{"1min":0.33800000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.51500000000000001,"15min":0.47199999999999998},"min":{"1min":0.33600000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.78600000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.51700000000000002}]},{"osd":2,"last update":"Fri Jul 14 17:42:54 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.54200000000000004,"15min":0.497},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.86099999999999999,"5min":0.879,"15min":0.879},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.52300000000000002,"15min":0.48899999999999999},"min":{"1min":0.36499999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.56299999999999994}]},{"osd":3,"last update":"Fri Jul 14 17:43:19 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53400000000000003,"15min":0.51600000000000001},"min":{"1min":0.31,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.75600000000000001,"5min":0.873,"15min":0.873},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.57699999999999996,"15min":0.54500000000000004},"min":{"1min":0.44700000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.752,"5min":0.873,"15min":0.873},"last":0.66500000000000004}]},{"osd":4,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.45400000000000001,"15min":0.42999999999999999},"min":{"1min":0.34699999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.45700000000000002,"15min":0.433},"min":{"1min":0.29399999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.79100000000000004}]},{"osd":5,"last update":"Fri Jul 14 17:42:59 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.60899999999999999,"15min":0.58499999999999996},"min":{"1min":0.48299999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.875},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.58199999999999996,"15min":0.57099999999999995},"min":{"1min":0.42699999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.85999999999999999}]},{"osd":6,"last update":"Fri Jul 14 17:43:25 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.41599999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.443,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.81899999999999995}]},{"osd":7,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.56899999999999995,"15min":0.52800000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.55300000000000005,"15min":0.53300000000000003},"min":{"1min":0.373,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.89000000000000001}]},{"osd":8,"last update":"Fri Jul 14 17:42:59 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.65900000000000003,"15min":0.65600000000000003},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1,"5min":1,"15min":1},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.70199999999999996,"15min":0.69499999999999995},"min":{"1min":0.54000000000000004,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.97099999999999997}]},{"osd":9,"last update":"Fri Jul 14 17:43:19 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65500000000000003,"15min":0.65700000000000003},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.88400000000000001,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70199999999999996,"15min":0.70099999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.94099999999999995}]},{"osd":10,"last update":"Fri Jul 14 17:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.747,"15min":0.73699999999999999},"min":{"1min":0.52600000000000002,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.71399999999999997,"15min":0.70099999999999996},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.999}]},{"osd":11,"last update":"Fri Jul 14 17:43:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.95999999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":483138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 17:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.61499999999999999,"15min":0.53700000000000003},"min":{"1min":0.38500000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":8.5199999999999996,"5min":8.5199999999999996,"15min":8.5199999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.0960000000000001,"5min":0.65800000000000003,"15min":0.56000000000000005},"min":{"1min":0.41599999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":10.31,"5min":10.31,"15min":10.31},"last":0.57899999999999996}]},{"osd":2,"last update":"Fri Jul 14 17:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55900000000000005,"15min":0.53400000000000003},"min":{"1min":0.40999999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.81299999999999994,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56999999999999995,"15min":0.55300000000000005},"min":{"1min":0.379,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.71099999999999997}]},{"osd":3,"last update":"Fri Jul 14 17:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":8.4629999999999992,"5min":8.4629999999999992,"15min":8.4629999999999992},"last":0.309},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":8.4900000000000002,"5min":8.4900000000000002,"15min":8.4900000000000002},"last":0.44600000000000001}]},{"osd":4,"last update":"Fri Jul 14 17:43:37 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55700000000000005,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.121,"5min":0.68100000000000005,"15min":0.58099999999999996},"min":{"1min":0.379,"5min":0.316,"15min":0.316},"max":{"1min":10.374000000000001,"5min":10.374000000000001,"15min":10.374000000000001},"last":0.67500000000000004}]},{"osd":5,"last update":"Fri Jul 14 17:42:58 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.56499999999999995,"15min":0.55100000000000005},"min":{"1min":0.45000000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.755,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59599999999999997,"15min":0.58999999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.84099999999999997,"5min":0.88,"15min":0.88},"last":0.497}]},{"osd":6,"last update":"Fri Jul 14 17:43:18 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.57999999999999996,"15min":0.502},"min":{"1min":0.30599999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":10.468999999999999,"5min":10.468999999999999,"15min":10.468999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.58499999999999996,"15min":0.51600000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":8.4350000000000005,"5min":8.4350000000000005,"15min":8.4350000000000005},"last":0.38200000000000001}]},{"osd":7,"last update":"Fri Jul 14 17:43:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":10.407999999999999,"5min":10.407999999999999,"15min":10.407999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.60299999999999998}]},{"osd":8,"last update":"Fri Jul 14 17:43:06 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.68500000000000005,"15min":0.69099999999999995},"min":{"1min":0.48999999999999999,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.82799999999999996,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.67100000000000004,"15min":0.67900000000000005},"min":{"1min":0.46200000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.89900000000000002,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.63700000000000001}]},{"osd":9,"last update":"Fri Jul 14 17:43:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.77600000000000002,"15min":0.72799999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":10.343,"5min":10.343,"15min":10.343},"last":0.40799999999999997},{"interface":"front","average":{"1min":1.5509999999999999,"5min":0.89000000000000001,"15min":0.78000000000000003},"min":{"1min":0.45300000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":18.739000000000001,"5min":18.739000000000001,"15min":18.739000000000001},"last":0.50700000000000001}]},{"osd":10,"last update":"Fri Jul 14 17:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.73299999999999998,"15min":0.74199999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.2090000000000001,"5min":0.85999999999999999,"15min":0.80200000000000005},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":10.439,"5min":10.439,"15min":10.439},"last":0.626}]},{"osd":11,"last update":"Fri Jul 14 17:43:06 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.68700000000000006}]}]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":541552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.47499999999999998},"min":{"1min":0.41099999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.48299999999999998,"15min":0.46700000000000003},"min":{"1min":0.39100000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.49299999999999999}]},{"osd":1,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.53100000000000003,"15min":0.48299999999999998},"min":{"1min":0.35199999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.55300000000000005,"15min":0.50900000000000001},"min":{"1min":0.5,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.56399999999999995}]},{"osd":3,"last update":"Fri Jul 14 17:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.57499999999999996,"15min":0.54700000000000004},"min":{"1min":0.38100000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55900000000000005,"15min":0.53900000000000003},"min":{"1min":0.40100000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.86899999999999999,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.54900000000000004}]},{"osd":4,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.59299999999999997}]},{"osd":5,"last update":"Fri Jul 14 17:42:57 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.45600000000000002,"15min":0.43099999999999999},"min":{"1min":0.30399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.45200000000000001,"15min":0.437},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.41499999999999998}]},{"osd":6,"last update":"Fri Jul 14 17:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.60099999999999998,"15min":0.58099999999999996},"min":{"1min":0.47199999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.76000000000000001,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59899999999999998,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81799999999999995,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.58199999999999996}]},{"osd":7,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.68600000000000005}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.61299999999999999,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.59299999999999997,"15min":0.57399999999999995},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.622}]},{"osd":9,"last update":"Fri Jul 14 17:43:27 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71399999999999997,"15min":0.70999999999999996},"min":{"1min":0.56999999999999995,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.76700000000000002,"15min":0.75600000000000001},"min":{"1min":0.64100000000000001,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":0.79200000000000004}]},{"osd":10,"last update":"Fri Jul 14 17:43:46 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.74099999999999999,"15min":0.73699999999999999},"min":{"1min":0.51300000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.77500000000000002,"15min":0.76300000000000001},"min":{"1min":0.59299999999999997,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.80900000000000005}]},{"osd":11,"last update":"Fri Jul 14 17:43:40 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.77500000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182461,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":481438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:16 2023","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.34300000000000003,"15min":0.32300000000000001},"min":{"1min":0.26900000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.437,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.36099999999999999,"15min":0.33200000000000002},"min":{"1min":0.28100000000000003,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.44800000000000001}]},{"osd":1,"last update":"Fri Jul 14 17:43:16 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54200000000000004,"15min":0.50600000000000001},"min":{"1min":0.38600000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.82599999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54200000000000004,"15min":0.505},"min":{"1min":0.33300000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.72499999999999998,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.51500000000000001}]},{"osd":2,"last update":"Fri Jul 14 17:43:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.54700000000000004,"15min":0.52400000000000002},"min":{"1min":0.41899999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.69699999999999995,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56499999999999995,"15min":0.52800000000000002},"min":{"1min":0.40899999999999997,"5min":0.379,"15min":0.379},"max":{"1min":0.71999999999999997,"5min":1.423,"15min":1.423},"last":0.89300000000000002}]},{"osd":4,"last update":"Fri Jul 14 17:43:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58299999999999996,"15min":0.56699999999999995},"min":{"1min":0.378,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.748,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61199999999999999,"15min":0.58499999999999996},"min":{"1min":0.42499999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79400000000000004,"5min":1.5589999999999999,"15min":1.5589999999999999},"last":0.72299999999999998}]},{"osd":5,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.38100000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85299999999999998,"5min":1.593,"15min":1.593},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.45400000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.85699999999999998,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.82999999999999996}]},{"osd":6,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.52600000000000002,"15min":0.52900000000000003},"min":{"1min":0.32800000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.73699999999999999,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.46899999999999997,"15min":0.47599999999999998},"min":{"1min":0.32700000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.57399999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.57099999999999995}]},{"osd":7,"last update":"Fri Jul 14 17:43:44 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.61899999999999999,"15min":0.63700000000000001},"min":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82999999999999996,"5min":1.28,"15min":1.28},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65500000000000003,"15min":0.64500000000000002},"min":{"1min":0.54400000000000004,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.82299999999999995,"5min":1.03,"15min":1.03},"last":0.874}]},{"osd":8,"last update":"Fri Jul 14 17:41:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.81299999999999994}]},{"osd":9,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.69299999999999995,"15min":0.70799999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.878,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.68100000000000005,"15min":0.69699999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.84999999999999998}]},{"osd":10,"last update":"Fri Jul 14 17:43:44 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.66200000000000003,"15min":0.67500000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.67700000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.751},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.749,"15min":0.76000000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":0.86799999999999999,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.91500000000000004}]},{"osd":11,"last update":"Fri Jul 14 17:43:44 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.999},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.333,"5min":1.333,"15min":1.333},"last":0.97299999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953527,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1003520,"data_stored":486146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.35399999999999998,"5min":0.311,"15min":0.311},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.52100000000000002,"15min":0.502},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.496}]},{"osd":1,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.41699999999999998,"15min":0.38200000000000001},"min":{"1min":0.372,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.61699999999999999,"5min":0.755,"15min":0.755},"last":0.443},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.42699999999999999,"15min":0.40000000000000002},"min":{"1min":0.38,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.621,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.49099999999999999}]},{"osd":2,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.57499999999999996,"15min":0.56299999999999994},"min":{"1min":0.496,"5min":0.38,"15min":0.38},"max":{"1min":0.80200000000000005,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.622,"15min":0.58199999999999996},"min":{"1min":0.54400000000000004,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89600000000000002,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.74399999999999999}]},{"osd":3,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.59799999999999998,"15min":0.55100000000000005},"min":{"1min":0.44600000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.86599999999999999,"5min":1.498,"15min":1.498},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.56200000000000006,"15min":0.53400000000000003},"min":{"1min":0.41399999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.81100000000000005,"5min":1.629,"15min":1.629},"last":0.53800000000000003}]},{"osd":5,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.63100000000000001,"15min":0.61099999999999999},"min":{"1min":0.51500000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89800000000000002,"5min":1.256,"15min":1.256},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.624,"15min":0.60999999999999999},"min":{"1min":0.44900000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.72199999999999998}]},{"osd":6,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.72199999999999998,"15min":0.746},"min":{"1min":0.47699999999999998,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.91800000000000004,"5min":2.7789999999999999,"15min":2.7789999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.68200000000000005,"15min":0.66700000000000004},"min":{"1min":0.46600000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.95899999999999996,"5min":1.476,"15min":1.476},"last":0.78900000000000003}]},{"osd":7,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.57899999999999996,"15min":0.53300000000000003},"min":{"1min":0.49099999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.97299999999999998,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.61399999999999999,"15min":0.58399999999999996},"min":{"1min":0.5,"5min":0.377,"15min":0.377},"max":{"1min":0.88500000000000001,"5min":1.002,"15min":1.002},"last":0.82199999999999995}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.65600000000000003,"15min":0.64400000000000002},"min":{"1min":0.41999999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72299999999999998,"15min":0.71399999999999997},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.77300000000000002}]},{"osd":9,"last update":"Fri Jul 14 17:43:20 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.69399999999999995,"15min":0.69099999999999995},"min":{"1min":0.54000000000000004,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.88100000000000001,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.80100000000000005,"15min":0.80400000000000005},"min":{"1min":0.61899999999999999,"5min":0.502,"15min":0.502},"max":{"1min":1.0049999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.83499999999999996}]},{"osd":10,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.82899999999999996,"15min":0.83099999999999996},"min":{"1min":0.63400000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0600000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.73199999999999998,"15min":0.72999999999999998},"min":{"1min":0.59799999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.73199999999999998}]},{"osd":11,"last update":"Fri Jul 14 17:43:41 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.80400000000000005}]}]},{"osd":5,"up_from":37,"seq":158913790004,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28608,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716544,"statfs":{"total":95995035648,"available":95965741056,"internally_reserved":0,"allocated":1622016,"data_stored":1078634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.47099999999999997,"15min":0.44700000000000001},"min":{"1min":0.35499999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77800000000000002,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.501,"5min":0.48299999999999998,"15min":0.45700000000000002},"min":{"1min":0.35599999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65400000000000003,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.61299999999999999}]},{"osd":1,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.55100000000000005,"15min":0.497},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":5.3630000000000004,"5min":5.3630000000000004,"15min":5.3630000000000004},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51300000000000001,"15min":0.48999999999999999},"min":{"1min":0.42399999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.72199999999999998}]},{"osd":2,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.33500000000000002},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.376,"15min":0.35999999999999999},"min":{"1min":0.30499999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.48299999999999998}]},{"osd":3,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.501,"15min":0.495},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.51800000000000002,"15min":0.495},"min":{"1min":0.38700000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.747,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.65000000000000002}]},{"osd":4,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.53500000000000003,"15min":0.50800000000000001},"min":{"1min":0.434,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.78100000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.54300000000000004,"15min":0.52000000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.63500000000000001}]},{"osd":6,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54400000000000004,"15min":0.54100000000000004},"min":{"1min":0.39000000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.75700000000000001,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.56699999999999995,"15min":0.54300000000000004},"min":{"1min":0.46400000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80800000000000005,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.69099999999999995}]},{"osd":7,"last update":"Fri Jul 14 17:43:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.58899999999999997,"15min":0.57399999999999995},"min":{"1min":0.47599999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.63300000000000001,"15min":0.61299999999999999},"min":{"1min":0.51200000000000001,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.79200000000000004}]},{"osd":8,"last update":"Fri Jul 14 17:43:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.54700000000000004,"15min":0.54000000000000004},"min":{"1min":0.38100000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.754,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.56899999999999995,"15min":0.56499999999999995},"min":{"1min":0.372,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.876,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.70499999999999996}]},{"osd":9,"last update":"Fri Jul 14 17:43:21 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.68799999999999994,"15min":0.68200000000000005},"min":{"1min":0.56999999999999995,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.91000000000000003,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68400000000000005,"15min":0.67600000000000005},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.81200000000000006}]},{"osd":10,"last update":"Fri Jul 14 17:43:47 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.69199999999999995,"15min":0.68200000000000005},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.88200000000000001,"15min":0.89400000000000002},"min":{"1min":0.623,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":0.94599999999999995,"5min":5.3330000000000002,"15min":5.3330000000000002},"last":0.90500000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":60629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T17:43:59.231 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-07-14T17:43:59.232 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-07-14T17:43:59.232 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-07-14T17:43:59.233 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph health --format=json 2023-07-14T17:43:59.234 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:58 smithi107 ceph-mon[128999]: pgmap v295: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2023-07-14T17:43:59.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:58 smithi178 ceph-mon[129368]: pgmap v295: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2023-07-14T17:43:59.785 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:44:00.188 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:43:59 smithi088 ceph-mon[127465]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:44:00.223 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:43:59 smithi107 ceph-mon[128999]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:44:00.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:43:59 smithi178 ceph-mon[129368]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T17:44:00.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.012+0000 7fb0511d0700 1 -- 172.21.15.88:0/3365796386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb04c103f10 msgr2=0x7fb04c10ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:00.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.012+0000 7fb0511d0700 1 --2- 172.21.15.88:0/3365796386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb04c103f10 0x7fb04c10ba90 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb04400b0d0 tx=0x7fb04400b3e0 comp rx=0 tx=0).stop 2023-07-14T17:44:00.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.012+0000 7fb0511d0700 1 -- 172.21.15.88:0/3365796386 shutdown_connections 2023-07-14T17:44:00.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.012+0000 7fb0511d0700 1 --2- 172.21.15.88:0/3365796386 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb04c103f10 0x7fb04c10ba90 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.012+0000 7fb0511d0700 1 --2- 172.21.15.88:0/3365796386 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb04c0fee70 0x7fb04c0ff240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.012+0000 7fb0511d0700 1 --2- 172.21.15.88:0/3365796386 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb04c0ff810 0x7fb04c1037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.012+0000 7fb0511d0700 1 -- 172.21.15.88:0/3365796386 >> 172.21.15.88:0/3365796386 conn(0x7fb04c075d20 msgr2=0x7fb04c076120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.012+0000 7fb0511d0700 1 -- 172.21.15.88:0/3365796386 shutdown_connections 2023-07-14T17:44:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.012+0000 7fb0511d0700 1 -- 172.21.15.88:0/3365796386 wait complete. 2023-07-14T17:44:00.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.013+0000 7fb0511d0700 1 Processor -- start 2023-07-14T17:44:00.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.013+0000 7fb0511d0700 1 -- start start 2023-07-14T17:44:00.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.013+0000 7fb0511d0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb04c0fee70 0x7fb04c194d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:00.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.013+0000 7fb0511d0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb04c0ff810 0x7fb04c1952a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:00.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.013+0000 7fb0511d0700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb04c103f10 0x7fb04c199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:00.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.013+0000 7fb0511d0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c199d20 con 0x7fb04c0ff810 2023-07-14T17:44:00.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.013+0000 7fb0511d0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c199e90 con 0x7fb04c0fee70 2023-07-14T17:44:00.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.013+0000 7fb0511d0700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb04c19a050 con 0x7fb04c103f10 2023-07-14T17:44:00.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.013+0000 7fb04a59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb04c0ff810 0x7fb04c1952a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:00.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.013+0000 7fb04a59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb04c0ff810 0x7fb04c1952a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42502/0 (socket says 172.21.15.88:42502) 2023-07-14T17:44:00.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.013+0000 7fb04a59c700 1 -- 172.21.15.88:0/3428937122 learned_addr learned my addr 172.21.15.88:0/3428937122 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:00.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.013+0000 7fb04ad9d700 1 --2- 172.21.15.88:0/3428937122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb04c0fee70 0x7fb04c194d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:00.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.013+0000 7fb04b59e700 1 --2- 172.21.15.88:0/3428937122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb04c103f10 0x7fb04c199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:00.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.014+0000 7fb04a59c700 1 -- 172.21.15.88:0/3428937122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb04c103f10 msgr2=0x7fb04c199710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:00.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.014+0000 7fb04a59c700 1 --2- 172.21.15.88:0/3428937122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb04c103f10 0x7fb04c199710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:00.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.014+0000 7fb04a59c700 1 -- 172.21.15.88:0/3428937122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb04c0fee70 msgr2=0x7fb04c194d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.014+0000 7fb04a59c700 1 --2- 172.21.15.88:0/3428937122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb04c0fee70 0x7fb04c194d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.014+0000 7fb04a59c700 1 -- 172.21.15.88:0/3428937122 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb044009d00 con 0x7fb04c0ff810 2023-07-14T17:44:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.014+0000 7fb04a59c700 1 --2- 172.21.15.88:0/3428937122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb04c0ff810 0x7fb04c1952a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb03400d980 tx=0x7fb03400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.014+0000 7fb043fff700 1 -- 172.21.15.88:0/3428937122 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb034009930 con 0x7fb04c0ff810 2023-07-14T17:44:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.014+0000 7fb043fff700 1 -- 172.21.15.88:0/3428937122 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb034010460 con 0x7fb04c0ff810 2023-07-14T17:44:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.014+0000 7fb0511d0700 1 -- 172.21.15.88:0/3428937122 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb04c19a330 con 0x7fb04c0ff810 2023-07-14T17:44:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.014+0000 7fb043fff700 1 -- 172.21.15.88:0/3428937122 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03400f620 con 0x7fb04c0ff810 2023-07-14T17:44:01.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.014+0000 7fb0511d0700 1 -- 172.21.15.88:0/3428937122 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb04c1a1b90 con 0x7fb04c0ff810 2023-07-14T17:44:01.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.015+0000 7fb0511d0700 1 -- 172.21.15.88:0/3428937122 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04c04e340 con 0x7fb04c0ff810 2023-07-14T17:44:01.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.016+0000 7fb043fff700 1 -- 172.21.15.88:0/3428937122 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb0340105d0 con 0x7fb04c0ff810 2023-07-14T17:44:01.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.016+0000 7fb043fff700 1 --2- 172.21.15.88:0/3428937122 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb0380a11e0 0x7fb0380a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:01.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.017+0000 7fb04ad9d700 1 --2- 172.21.15.88:0/3428937122 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb0380a11e0 0x7fb0380a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:01.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.017+0000 7fb043fff700 1 -- 172.21.15.88:0/3428937122 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb0340cc800 con 0x7fb04c0ff810 2023-07-14T17:44:01.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.017+0000 7fb04ad9d700 1 --2- 172.21.15.88:0/3428937122 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb0380a11e0 0x7fb0380a3690 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb03c005950 tx=0x7fb03c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:01.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.021+0000 7fb043fff700 1 -- 172.21.15.88:0/3428937122 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb034095ae0 con 0x7fb04c0ff810 2023-07-14T17:44:01.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:00 smithi088 ceph-mon[127465]: pgmap v296: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T17:44:01.223 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:00 smithi107 ceph-mon[128999]: pgmap v296: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T17:44:01.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.245+0000 7fb0511d0700 1 -- 172.21.15.88:0/3428937122 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb04c1a1d20 con 0x7fb04c0ff810 2023-07-14T17:44:01.227 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.246+0000 7fb043fff700 1 -- 172.21.15.88:0/3428937122 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fb034099100 con 0x7fb04c0ff810 2023-07-14T17:44:01.227 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:01.228 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-07-14T17:44:01.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.248+0000 7fb0511d0700 1 -- 172.21.15.88:0/3428937122 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb0380a11e0 msgr2=0x7fb0380a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:01.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.248+0000 7fb0511d0700 1 --2- 172.21.15.88:0/3428937122 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb0380a11e0 0x7fb0380a3690 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb03c005950 tx=0x7fb03c00a300 comp rx=0 tx=0).stop 2023-07-14T17:44:01.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.248+0000 7fb0511d0700 1 -- 172.21.15.88:0/3428937122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb04c0ff810 msgr2=0x7fb04c1952a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:01.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.248+0000 7fb0511d0700 1 --2- 172.21.15.88:0/3428937122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb04c0ff810 0x7fb04c1952a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb03400d980 tx=0x7fb03400dc90 comp rx=0 tx=0).stop 2023-07-14T17:44:01.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.248+0000 7fb0511d0700 1 -- 172.21.15.88:0/3428937122 shutdown_connections 2023-07-14T17:44:01.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.248+0000 7fb0511d0700 1 --2- 172.21.15.88:0/3428937122 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb04c103f10 0x7fb04c199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:01.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.248+0000 7fb0511d0700 1 --2- 172.21.15.88:0/3428937122 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb04c0fee70 0x7fb04c194d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:01.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.248+0000 7fb0511d0700 1 --2- 172.21.15.88:0/3428937122 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb0380a11e0 0x7fb0380a3690 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:01.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.248+0000 7fb0511d0700 1 --2- 172.21.15.88:0/3428937122 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb04c0ff810 0x7fb04c1952a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:01.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.248+0000 7fb0511d0700 1 -- 172.21.15.88:0/3428937122 >> 172.21.15.88:0/3428937122 conn(0x7fb04c075d20 msgr2=0x7fb04c0fc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:01.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.248+0000 7fb0511d0700 1 -- 172.21.15.88:0/3428937122 shutdown_connections 2023-07-14T17:44:01.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.248+0000 7fb0511d0700 1 -- 172.21.15.88:0/3428937122 wait complete. 2023-07-14T17:44:01.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:00 smithi178 ceph-mon[129368]: pgmap v296: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T17:44:01.632 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-07-14T17:44:01.633 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:44:01.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.777+0000 7fcfd2077700 1 -- 172.21.15.88:0/2925837065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc1006c0 msgr2=0x7fcfcc106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:01.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.777+0000 7fcfd2077700 1 --2- 172.21.15.88:0/2925837065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc1006c0 0x7fcfcc106f80 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400b0a0 tx=0x7fcfc400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:01.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.777+0000 7fcfd2077700 1 -- 172.21.15.88:0/2925837065 shutdown_connections 2023-07-14T17:44:01.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.777+0000 7fcfd2077700 1 --2- 172.21.15.88:0/2925837065 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc1006c0 0x7fcfcc106f80 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:01.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.777+0000 7fcfd2077700 1 --2- 172.21.15.88:0/2925837065 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfcc0feb40 0x7fcfcc0fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:01.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.777+0000 7fcfd2077700 1 --2- 172.21.15.88:0/2925837065 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfcc0ffd30 0x7fcfcc100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:01.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.777+0000 7fcfd2077700 1 -- 172.21.15.88:0/2925837065 >> 172.21.15.88:0/2925837065 conn(0x7fcfcc0fa160 msgr2=0x7fcfcc0fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:01.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.777+0000 7fcfd2077700 1 -- 172.21.15.88:0/2925837065 shutdown_connections 2023-07-14T17:44:01.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.777+0000 7fcfd2077700 1 -- 172.21.15.88:0/2925837065 wait complete. 2023-07-14T17:44:01.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.777+0000 7fcfd2077700 1 Processor -- start 2023-07-14T17:44:01.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.777+0000 7fcfd2077700 1 -- start start 2023-07-14T17:44:01.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfd2077700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfcc0feb40 0x7fcfcc1959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:01.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfd2077700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfcc0ffd30 0x7fcfcc195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:01.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfd2077700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc1006c0 0x7fcfcc19cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:01.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfd2077700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc19d540 con 0x7fcfcc1006c0 2023-07-14T17:44:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfd2077700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc19d680 con 0x7fcfcc0ffd30 2023-07-14T17:44:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfd2077700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfcc19d810 con 0x7fcfcc0feb40 2023-07-14T17:44:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfcbfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc1006c0 0x7fcfcc19cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfcbfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc1006c0 0x7fcfcc19cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42518/0 (socket says 172.21.15.88:42518) 2023-07-14T17:44:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfcbfff700 1 -- 172.21.15.88:0/3680579941 learned_addr learned my addr 172.21.15.88:0/3680579941 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfcaffd700 1 --2- 172.21.15.88:0/3680579941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfcc0ffd30 0x7fcfcc195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:01.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfcb7fe700 1 --2- 172.21.15.88:0/3680579941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfcc0feb40 0x7fcfcc1959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfcbfff700 1 -- 172.21.15.88:0/3680579941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfcc0feb40 msgr2=0x7fcfcc1959c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfcbfff700 1 --2- 172.21.15.88:0/3680579941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfcc0feb40 0x7fcfcc1959c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfcbfff700 1 -- 172.21.15.88:0/3680579941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfcc0ffd30 msgr2=0x7fcfcc195f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfcbfff700 1 --2- 172.21.15.88:0/3680579941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfcc0ffd30 0x7fcfcc195f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfcbfff700 1 -- 172.21.15.88:0/3680579941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc4009d00 con 0x7fcfcc1006c0 2023-07-14T17:44:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfcbfff700 1 --2- 172.21.15.88:0/3680579941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc1006c0 0x7fcfcc19cf70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4009fd0 tx=0x7fcfc400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfc8ff9700 1 -- 172.21.15.88:0/3680579941 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc400e040 con 0x7fcfcc1006c0 2023-07-14T17:44:01.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfc8ff9700 1 -- 172.21.15.88:0/3680579941 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcfc4007760 con 0x7fcfcc1006c0 2023-07-14T17:44:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfd2077700 1 -- 172.21.15.88:0/3680579941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfcc19da90 con 0x7fcfcc1006c0 2023-07-14T17:44:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfc8ff9700 1 -- 172.21.15.88:0/3680579941 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc401b6c0 con 0x7fcfcc1006c0 2023-07-14T17:44:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.778+0000 7fcfd2077700 1 -- 172.21.15.88:0/3680579941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfcc19df30 con 0x7fcfcc1006c0 2023-07-14T17:44:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.779+0000 7fcfd2077700 1 -- 172.21.15.88:0/3680579941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfcc10a290 con 0x7fcfcc1006c0 2023-07-14T17:44:01.765 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.780+0000 7fcfc8ff9700 1 -- 172.21.15.88:0/3680579941 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fcfc401b820 con 0x7fcfcc1006c0 2023-07-14T17:44:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.781+0000 7fcfc8ff9700 1 --2- 172.21.15.88:0/3680579941 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcfb40a1220 0x7fcfb40a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.781+0000 7fcfcb7fe700 1 --2- 172.21.15.88:0/3680579941 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcfb40a1220 0x7fcfb40a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.781+0000 7fcfc8ff9700 1 -- 172.21.15.88:0/3680579941 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcfc40ce660 con 0x7fcfcc1006c0 2023-07-14T17:44:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.781+0000 7fcfcb7fe700 1 --2- 172.21.15.88:0/3680579941 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcfb40a1220 0x7fcfb40a36d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0ffb90 tx=0x7fcfbc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:01.766 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.784+0000 7fcfc8ff9700 1 -- 172.21.15.88:0/3680579941 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcfc4097970 con 0x7fcfcc1006c0 2023-07-14T17:44:01.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.983+0000 7fcfd2077700 1 -- 172.21.15.88:0/3680579941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcfcc19e540 con 0x7fcfcc1006c0 2023-07-14T17:44:01.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.985+0000 7fcfc8ff9700 1 -- 172.21.15.88:0/3680579941 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24512 (secure 0 0 0) 0x7fcfc409af90 con 0x7fcfcc1006c0 2023-07-14T17:44:01.967 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:01.967 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":88,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:43:44.887959+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:44:01.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.987+0000 7fcfd2077700 1 -- 172.21.15.88:0/3680579941 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcfb40a1220 msgr2=0x7fcfb40a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:01.968 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.987+0000 7fcfd2077700 1 --2- 172.21.15.88:0/3680579941 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcfb40a1220 0x7fcfb40a36d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc0ffb90 tx=0x7fcfbc009260 comp rx=0 tx=0).stop 2023-07-14T17:44:01.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.987+0000 7fcfd2077700 1 -- 172.21.15.88:0/3680579941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc1006c0 msgr2=0x7fcfcc19cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:01.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.987+0000 7fcfd2077700 1 --2- 172.21.15.88:0/3680579941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc1006c0 0x7fcfcc19cf70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4009fd0 tx=0x7fcfc400bc80 comp rx=0 tx=0).stop 2023-07-14T17:44:01.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.987+0000 7fcfd2077700 1 -- 172.21.15.88:0/3680579941 shutdown_connections 2023-07-14T17:44:01.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.987+0000 7fcfd2077700 1 --2- 172.21.15.88:0/3680579941 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcfb40a1220 0x7fcfb40a36d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:01.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.987+0000 7fcfd2077700 1 --2- 172.21.15.88:0/3680579941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcfcc1006c0 0x7fcfcc19cf70 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:01.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.987+0000 7fcfd2077700 1 --2- 172.21.15.88:0/3680579941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcfcc0feb40 0x7fcfcc1959c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:01.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.987+0000 7fcfd2077700 1 --2- 172.21.15.88:0/3680579941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcfcc0ffd30 0x7fcfcc195f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:01.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.987+0000 7fcfd2077700 1 -- 172.21.15.88:0/3680579941 >> 172.21.15.88:0/3680579941 conn(0x7fcfcc0fa160 msgr2=0x7fcfcc0fa8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:01.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.987+0000 7fcfd2077700 1 -- 172.21.15.88:0/3680579941 shutdown_connections 2023-07-14T17:44:01.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:01.987+0000 7fcfd2077700 1 -- 172.21.15.88:0/3680579941 wait complete. 2023-07-14T17:44:01.981 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T17:38:50.126535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T17:42:40.725870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T17:42:41.512749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T17:42:44.764573+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T17:44:01.982 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T17:44:02.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.124+0000 7f3bef5cc700 1 -- 172.21.15.88:0/3057160918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3be8100690 msgr2=0x7f3be8106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:02.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.124+0000 7f3bef5cc700 1 --2- 172.21.15.88:0/3057160918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3be8100690 0x7f3be8106f50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3be000b0a0 tx=0x7f3be000b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.106 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.124+0000 7f3bef5cc700 1 -- 172.21.15.88:0/3057160918 shutdown_connections 2023-07-14T17:44:02.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.124+0000 7f3bef5cc700 1 --2- 172.21.15.88:0/3057160918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3be8100690 0x7f3be8106f50 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.124+0000 7f3bef5cc700 1 --2- 172.21.15.88:0/3057160918 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3be80feb10 0x7f3be80fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.124+0000 7f3bef5cc700 1 --2- 172.21.15.88:0/3057160918 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3be80ffd00 0x7f3be8100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.124+0000 7f3bef5cc700 1 -- 172.21.15.88:0/3057160918 >> 172.21.15.88:0/3057160918 conn(0x7f3be80fa130 msgr2=0x7f3be80fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:02.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.124+0000 7f3bef5cc700 1 -- 172.21.15.88:0/3057160918 shutdown_connections 2023-07-14T17:44:02.107 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.124+0000 7f3bef5cc700 1 -- 172.21.15.88:0/3057160918 wait complete. 2023-07-14T17:44:02.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.125+0000 7f3bef5cc700 1 Processor -- start 2023-07-14T17:44:02.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.125+0000 7f3bef5cc700 1 -- start start 2023-07-14T17:44:02.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.125+0000 7f3bef5cc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3be80feb10 0x7f3be8194910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:02.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.125+0000 7f3bef5cc700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3be80ffd00 0x7f3be8194e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:02.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.125+0000 7f3bef5cc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3be8100690 0x7f3be819bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:02.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.125+0000 7f3bef5cc700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be819c490 con 0x7f3be8100690 2023-07-14T17:44:02.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.125+0000 7f3bef5cc700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be819c5d0 con 0x7f3be80feb10 2023-07-14T17:44:02.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.125+0000 7f3bef5cc700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be819c760 con 0x7f3be80ffd00 2023-07-14T17:44:02.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.125+0000 7f3bedb69700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3be8100690 0x7f3be819bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:02.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.125+0000 7f3bedb69700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3be8100690 0x7f3be819bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42532/0 (socket says 172.21.15.88:42532) 2023-07-14T17:44:02.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.125+0000 7f3bedb69700 1 -- 172.21.15.88:0/3751378170 learned_addr learned my addr 172.21.15.88:0/3751378170 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:02.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.125+0000 7f3bed368700 1 --2- 172.21.15.88:0/3751378170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3be80feb10 0x7f3be8194910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:02.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.125+0000 7f3becb67700 1 --2- 172.21.15.88:0/3751378170 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3be80ffd00 0x7f3be8194e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:02.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.125+0000 7f3bedb69700 1 -- 172.21.15.88:0/3751378170 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3be80ffd00 msgr2=0x7f3be8194e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:02.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.125+0000 7f3bedb69700 1 --2- 172.21.15.88:0/3751378170 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3be80ffd00 0x7f3be8194e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.125+0000 7f3bedb69700 1 -- 172.21.15.88:0/3751378170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3be80feb10 msgr2=0x7f3be8194910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:02.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.125+0000 7f3bedb69700 1 --2- 172.21.15.88:0/3751378170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3be80feb10 0x7f3be8194910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.126+0000 7f3bedb69700 1 -- 172.21.15.88:0/3751378170 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be0009d00 con 0x7f3be8100690 2023-07-14T17:44:02.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.126+0000 7f3becb67700 1 --2- 172.21.15.88:0/3751378170 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3be80ffd00 0x7f3be8194e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:02.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.126+0000 7f3bedb69700 1 --2- 172.21.15.88:0/3751378170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3be8100690 0x7f3be819bec0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3be000b830 tx=0x7f3be0004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:02.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.126+0000 7f3bde7fc700 1 -- 172.21.15.88:0/3751378170 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be000e040 con 0x7f3be8100690 2023-07-14T17:44:02.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.126+0000 7f3bef5cc700 1 -- 172.21.15.88:0/3751378170 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be819c9e0 con 0x7f3be8100690 2023-07-14T17:44:02.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.126+0000 7f3bde7fc700 1 -- 172.21.15.88:0/3751378170 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3be0009440 con 0x7f3be8100690 2023-07-14T17:44:02.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.126+0000 7f3bde7fc700 1 -- 172.21.15.88:0/3751378170 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be001dbe0 con 0x7f3be8100690 2023-07-14T17:44:02.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.126+0000 7f3bef5cc700 1 -- 172.21.15.88:0/3751378170 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be819ce60 con 0x7f3be8100690 2023-07-14T17:44:02.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.127+0000 7f3bef5cc700 1 -- 172.21.15.88:0/3751378170 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be8106100 con 0x7f3be8100690 2023-07-14T17:44:02.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.128+0000 7f3bde7fc700 1 -- 172.21.15.88:0/3751378170 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3be0005450 con 0x7f3be8100690 2023-07-14T17:44:02.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.128+0000 7f3bde7fc700 1 --2- 172.21.15.88:0/3751378170 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3bd40a11e0 0x7f3bd40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:02.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.128+0000 7f3bed368700 1 --2- 172.21.15.88:0/3751378170 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3bd40a11e0 0x7f3bd40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:02.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.128+0000 7f3bde7fc700 1 -- 172.21.15.88:0/3751378170 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3be00ceb40 con 0x7f3be8100690 2023-07-14T17:44:02.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.129+0000 7f3bed368700 1 --2- 172.21.15.88:0/3751378170 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3bd40a11e0 0x7f3bd40a3690 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8005fd0 tx=0x7f3bd8005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:02.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.132+0000 7f3bde7fc700 1 -- 172.21.15.88:0/3751378170 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3be0097eb0 con 0x7f3be8100690 2023-07-14T17:44:02.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:01 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3428937122' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T17:44:02.223 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:01 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3428937122' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T17:44:02.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.319+0000 7f3bef5cc700 1 -- 172.21.15.88:0/3751378170 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3be802ca70 con 0x7f3be8100690 2023-07-14T17:44:02.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.319+0000 7f3bde7fc700 1 -- 172.21.15.88:0/3751378170 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3be009b4d0 con 0x7f3be8100690 2023-07-14T17:44:02.301 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-07-14T17:44:02.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.321+0000 7f3bef5cc700 1 -- 172.21.15.88:0/3751378170 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3bd40a11e0 msgr2=0x7f3bd40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:02.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.321+0000 7f3bef5cc700 1 --2- 172.21.15.88:0/3751378170 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3bd40a11e0 0x7f3bd40a3690 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8005fd0 tx=0x7f3bd8005dd0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.321+0000 7f3bef5cc700 1 -- 172.21.15.88:0/3751378170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3be8100690 msgr2=0x7f3be819bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:02.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.321+0000 7f3bef5cc700 1 --2- 172.21.15.88:0/3751378170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3be8100690 0x7f3be819bec0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3be000b830 tx=0x7f3be0004d90 comp rx=0 tx=0).stop 2023-07-14T17:44:02.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.321+0000 7f3bef5cc700 1 -- 172.21.15.88:0/3751378170 shutdown_connections 2023-07-14T17:44:02.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.321+0000 7f3bef5cc700 1 --2- 172.21.15.88:0/3751378170 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3bd40a11e0 0x7f3bd40a3690 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.321+0000 7f3bef5cc700 1 --2- 172.21.15.88:0/3751378170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3be8100690 0x7f3be819bec0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.321+0000 7f3bef5cc700 1 --2- 172.21.15.88:0/3751378170 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3be80feb10 0x7f3be8194910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.321+0000 7f3bef5cc700 1 --2- 172.21.15.88:0/3751378170 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3be80ffd00 0x7f3be8194e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.321+0000 7f3bef5cc700 1 -- 172.21.15.88:0/3751378170 >> 172.21.15.88:0/3751378170 conn(0x7f3be80fa130 msgr2=0x7f3be8103240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:02.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.322+0000 7f3bef5cc700 1 -- 172.21.15.88:0/3751378170 shutdown_connections 2023-07-14T17:44:02.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.322+0000 7f3bef5cc700 1 -- 172.21.15.88:0/3751378170 wait complete. 2023-07-14T17:44:02.315 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T17:44:02.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:01 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3428937122' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T17:44:02.439 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.458+0000 7f781ba7f700 1 -- 172.21.15.88:0/1595095503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7814100690 msgr2=0x7f7814106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:02.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.458+0000 7f781ba7f700 1 --2- 172.21.15.88:0/1595095503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7814100690 0x7f7814106f50 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f780c00b0a0 tx=0x7f780c00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.458+0000 7f781ba7f700 1 -- 172.21.15.88:0/1595095503 shutdown_connections 2023-07-14T17:44:02.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.458+0000 7f781ba7f700 1 --2- 172.21.15.88:0/1595095503 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7814100690 0x7f7814106f50 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.440 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.458+0000 7f781ba7f700 1 --2- 172.21.15.88:0/1595095503 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78140feb10 0x7f78140fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.458+0000 7f781ba7f700 1 --2- 172.21.15.88:0/1595095503 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78140ffd00 0x7f7814100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.458+0000 7f781ba7f700 1 -- 172.21.15.88:0/1595095503 >> 172.21.15.88:0/1595095503 conn(0x7f78140fa130 msgr2=0x7f78140fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:02.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.458+0000 7f781ba7f700 1 -- 172.21.15.88:0/1595095503 shutdown_connections 2023-07-14T17:44:02.441 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.458+0000 7f781ba7f700 1 -- 172.21.15.88:0/1595095503 wait complete. 2023-07-14T17:44:02.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.458+0000 7f781ba7f700 1 Processor -- start 2023-07-14T17:44:02.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781ba7f700 1 -- start start 2023-07-14T17:44:02.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781ba7f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78140feb10 0x7f78141947f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:02.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781ba7f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78140ffd00 0x7f7814194d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:02.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781ba7f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7814100690 0x7f781419bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:02.443 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781ba7f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781419c370 con 0x7f7814100690 2023-07-14T17:44:02.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781ba7f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781419c4b0 con 0x7f78140ffd00 2023-07-14T17:44:02.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781ba7f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781419c640 con 0x7f78140feb10 2023-07-14T17:44:02.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781a01c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7814100690 0x7f781419bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:02.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781a01c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7814100690 0x7f781419bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42546/0 (socket says 172.21.15.88:42546) 2023-07-14T17:44:02.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781a01c700 1 -- 172.21.15.88:0/1499237755 learned_addr learned my addr 172.21.15.88:0/1499237755 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:02.444 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781981b700 1 --2- 172.21.15.88:0/1499237755 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78140feb10 0x7f78141947f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:02.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781901a700 1 --2- 172.21.15.88:0/1499237755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78140ffd00 0x7f7814194d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:02.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781a01c700 1 -- 172.21.15.88:0/1499237755 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78140feb10 msgr2=0x7f78141947f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:02.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781a01c700 1 --2- 172.21.15.88:0/1499237755 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78140feb10 0x7f78141947f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781a01c700 1 -- 172.21.15.88:0/1499237755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78140ffd00 msgr2=0x7f7814194d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:02.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781a01c700 1 --2- 172.21.15.88:0/1499237755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78140ffd00 0x7f7814194d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781a01c700 1 -- 172.21.15.88:0/1499237755 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f780c009d00 con 0x7f7814100690 2023-07-14T17:44:02.445 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781901a700 1 --2- 172.21.15.88:0/1499237755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78140ffd00 0x7f7814194d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:02.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781a01c700 1 --2- 172.21.15.88:0/1499237755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7814100690 0x7f781419bda0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f780c00b830 tx=0x7f780c004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:02.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f780affd700 1 -- 172.21.15.88:0/1499237755 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780c00e040 con 0x7f7814100690 2023-07-14T17:44:02.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f781ba7f700 1 -- 172.21.15.88:0/1499237755 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781419c8c0 con 0x7f7814100690 2023-07-14T17:44:02.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f780affd700 1 -- 172.21.15.88:0/1499237755 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f780c009440 con 0x7f7814100690 2023-07-14T17:44:02.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.459+0000 7f780affd700 1 -- 172.21.15.88:0/1499237755 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f780c01dc00 con 0x7f7814100690 2023-07-14T17:44:02.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.460+0000 7f781ba7f700 1 -- 172.21.15.88:0/1499237755 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f781419cce0 con 0x7f7814100690 2023-07-14T17:44:02.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.460+0000 7f781ba7f700 1 -- 172.21.15.88:0/1499237755 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f781410a1a0 con 0x7f7814100690 2023-07-14T17:44:02.446 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.461+0000 7f780affd700 1 -- 172.21.15.88:0/1499237755 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f780c005450 con 0x7f7814100690 2023-07-14T17:44:02.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.462+0000 7f780affd700 1 --2- 172.21.15.88:0/1499237755 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f78000a11e0 0x7f78000a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:02.447 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.462+0000 7f781981b700 1 --2- 172.21.15.88:0/1499237755 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f78000a11e0 0x7f78000a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:02.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.462+0000 7f780affd700 1 -- 172.21.15.88:0/1499237755 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f780c0ceb20 con 0x7f7814100690 2023-07-14T17:44:02.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.462+0000 7f781981b700 1 --2- 172.21.15.88:0/1499237755 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f78000a11e0 0x7f78000a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f78140ffb60 tx=0x7f780400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:02.448 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.466+0000 7f780affd700 1 -- 172.21.15.88:0/1499237755 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f780c097e90 con 0x7f7814100690 2023-07-14T17:44:02.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.658+0000 7f781ba7f700 1 -- 172.21.15.88:0/1499237755 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f78141059c0 con 0x7f7814100690 2023-07-14T17:44:02.640 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.658+0000 7f780affd700 1 -- 172.21.15.88:0/1499237755 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f780c09b4b0 con 0x7f7814100690 2023-07-14T17:44:02.640 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-07-14T17:44:02.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.660+0000 7f781ba7f700 1 -- 172.21.15.88:0/1499237755 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f78000a11e0 msgr2=0x7f78000a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:02.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.660+0000 7f781ba7f700 1 --2- 172.21.15.88:0/1499237755 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f78000a11e0 0x7f78000a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f78140ffb60 tx=0x7f780400b410 comp rx=0 tx=0).stop 2023-07-14T17:44:02.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.660+0000 7f781ba7f700 1 -- 172.21.15.88:0/1499237755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7814100690 msgr2=0x7f781419bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:02.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.660+0000 7f781ba7f700 1 --2- 172.21.15.88:0/1499237755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7814100690 0x7f781419bda0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f780c00b830 tx=0x7f780c004ce0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.660+0000 7f781ba7f700 1 -- 172.21.15.88:0/1499237755 shutdown_connections 2023-07-14T17:44:02.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.660+0000 7f781ba7f700 1 --2- 172.21.15.88:0/1499237755 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f78000a11e0 0x7f78000a3690 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.660+0000 7f781ba7f700 1 --2- 172.21.15.88:0/1499237755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7814100690 0x7f781419bda0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.660+0000 7f781ba7f700 1 --2- 172.21.15.88:0/1499237755 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f78140feb10 0x7f78141947f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.660+0000 7f781ba7f700 1 --2- 172.21.15.88:0/1499237755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f78140ffd00 0x7f7814194d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.660+0000 7f781ba7f700 1 -- 172.21.15.88:0/1499237755 >> 172.21.15.88:0/1499237755 conn(0x7f78140fa130 msgr2=0x7f78140fc340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:02.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.661+0000 7f781ba7f700 1 -- 172.21.15.88:0/1499237755 shutdown_connections 2023-07-14T17:44:02.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.661+0000 7f781ba7f700 1 -- 172.21.15.88:0/1499237755 wait complete. 2023-07-14T17:44:02.654 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T17:44:02.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.797+0000 7ff8f2658700 1 -- 172.21.15.88:0/2111260374 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8ec0ff7b0 msgr2=0x7ff8ec107060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:02.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.797+0000 7ff8f2658700 1 --2- 172.21.15.88:0/2111260374 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8ec0ff7b0 0x7ff8ec107060 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff8e400b0d0 tx=0x7ff8e400b3e0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.797+0000 7ff8f2658700 1 -- 172.21.15.88:0/2111260374 shutdown_connections 2023-07-14T17:44:02.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.797+0000 7ff8f2658700 1 --2- 172.21.15.88:0/2111260374 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8ec0ff7b0 0x7ff8ec107060 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.779 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.797+0000 7ff8f2658700 1 --2- 172.21.15.88:0/2111260374 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ec100110 0x7ff8ec100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.797+0000 7ff8f2658700 1 --2- 172.21.15.88:0/2111260374 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8ec0fee00 0x7ff8ec0ff270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.797+0000 7ff8f2658700 1 -- 172.21.15.88:0/2111260374 >> 172.21.15.88:0/2111260374 conn(0x7ff8ec0f98a0 msgr2=0x7ff8ec0fbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:02.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.797+0000 7ff8f2658700 1 -- 172.21.15.88:0/2111260374 shutdown_connections 2023-07-14T17:44:02.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.797+0000 7ff8f2658700 1 -- 172.21.15.88:0/2111260374 wait complete. 2023-07-14T17:44:02.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.797+0000 7ff8f2658700 1 Processor -- start 2023-07-14T17:44:02.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.797+0000 7ff8f2658700 1 -- start start 2023-07-14T17:44:02.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8f2658700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ec0fee00 0x7ff8ec195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:02.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8f2658700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8ec0ff7b0 0x7ff8ec195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:02.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8f2658700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8ec100110 0x7ff8ec19cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:02.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8f2658700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec19d510 con 0x7ff8ec0ff7b0 2023-07-14T17:44:02.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8f2658700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec19d650 con 0x7ff8ec0fee00 2023-07-14T17:44:02.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8f2658700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ec19d7e0 con 0x7ff8ec100110 2023-07-14T17:44:02.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8eb7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8ec0ff7b0 0x7ff8ec195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:02.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8eb7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8ec0ff7b0 0x7ff8ec195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42558/0 (socket says 172.21.15.88:42558) 2023-07-14T17:44:02.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8eb7fe700 1 -- 172.21.15.88:0/1717549957 learned_addr learned my addr 172.21.15.88:0/1717549957 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:02.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8f0bf5700 1 --2- 172.21.15.88:0/1717549957 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8ec100110 0x7ff8ec19cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:02.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8ebfff700 1 --2- 172.21.15.88:0/1717549957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ec0fee00 0x7ff8ec195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:02.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8eb7fe700 1 -- 172.21.15.88:0/1717549957 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8ec100110 msgr2=0x7ff8ec19cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:02.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8eb7fe700 1 --2- 172.21.15.88:0/1717549957 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8ec100110 0x7ff8ec19cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8eb7fe700 1 -- 172.21.15.88:0/1717549957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ec0fee00 msgr2=0x7ff8ec195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:02.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8eb7fe700 1 --2- 172.21.15.88:0/1717549957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ec0fee00 0x7ff8ec195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.784 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8eb7fe700 1 -- 172.21.15.88:0/1717549957 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e4009d00 con 0x7ff8ec0ff7b0 2023-07-14T17:44:02.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8ebfff700 1 --2- 172.21.15.88:0/1717549957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ec0fee00 0x7ff8ec195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:02.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8eb7fe700 1 --2- 172.21.15.88:0/1717549957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8ec0ff7b0 0x7ff8ec195ed0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc00d950 tx=0x7ff8dc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:02.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8e97fa700 1 -- 172.21.15.88:0/1717549957 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc009940 con 0x7ff8ec0ff7b0 2023-07-14T17:44:02.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8f2658700 1 -- 172.21.15.88:0/1717549957 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ec19dac0 con 0x7ff8ec0ff7b0 2023-07-14T17:44:02.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8e97fa700 1 -- 172.21.15.88:0/1717549957 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff8dc010460 con 0x7ff8ec0ff7b0 2023-07-14T17:44:02.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.798+0000 7ff8e97fa700 1 -- 172.21.15.88:0/1717549957 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc00f620 con 0x7ff8ec0ff7b0 2023-07-14T17:44:02.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.799+0000 7ff8f2658700 1 -- 172.21.15.88:0/1717549957 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ec19e020 con 0x7ff8ec0ff7b0 2023-07-14T17:44:02.785 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.799+0000 7ff8f2658700 1 -- 172.21.15.88:0/1717549957 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ec10a210 con 0x7ff8ec0ff7b0 2023-07-14T17:44:02.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.801+0000 7ff8e97fa700 1 -- 172.21.15.88:0/1717549957 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff8dc0105d0 con 0x7ff8ec0ff7b0 2023-07-14T17:44:02.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.801+0000 7ff8e97fa700 1 --2- 172.21.15.88:0/1717549957 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff8d40a11e0 0x7ff8d40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:02.786 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.801+0000 7ff8e97fa700 1 -- 172.21.15.88:0/1717549957 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff8dc0ccbd0 con 0x7ff8ec0ff7b0 2023-07-14T17:44:02.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.801+0000 7ff8ebfff700 1 --2- 172.21.15.88:0/1717549957 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff8d40a11e0 0x7ff8d40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:02.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.801+0000 7ff8ebfff700 1 --2- 172.21.15.88:0/1717549957 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff8d40a11e0 0x7ff8d40a3690 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff8e000bc10 tx=0x7ff8e000b5f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:02.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.804+0000 7ff8e97fa700 1 -- 172.21.15.88:0/1717549957 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff8dc095eb0 con 0x7ff8ec0ff7b0 2023-07-14T17:44:02.938 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:02 smithi088 ceph-mon[127465]: pgmap v297: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2023-07-14T17:44:02.939 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:02 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3680579941' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:02.939 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:02 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3751378170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:44:02.939 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:02 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1499237755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:44:02.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.994+0000 7ff8f2658700 1 -- 172.21.15.88:0/1717549957 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff8ec10a3d0 con 0x7ff8ec0ff7b0 2023-07-14T17:44:02.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.994+0000 7ff8e97fa700 1 -- 172.21.15.88:0/1717549957 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff8dc0200a0 con 0x7ff8ec0ff7b0 2023-07-14T17:44:02.976 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 64 2023-07-14T17:44:02.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.996+0000 7ff8f2658700 1 -- 172.21.15.88:0/1717549957 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff8d40a11e0 msgr2=0x7ff8d40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:02.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.996+0000 7ff8f2658700 1 --2- 172.21.15.88:0/1717549957 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff8d40a11e0 0x7ff8d40a3690 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff8e000bc10 tx=0x7ff8e000b5f0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.996+0000 7ff8f2658700 1 -- 172.21.15.88:0/1717549957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8ec0ff7b0 msgr2=0x7ff8ec195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:02.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.996+0000 7ff8f2658700 1 --2- 172.21.15.88:0/1717549957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8ec0ff7b0 0x7ff8ec195ed0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc00d950 tx=0x7ff8dc00dc60 comp rx=0 tx=0).stop 2023-07-14T17:44:02.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.997+0000 7ff8f2658700 1 -- 172.21.15.88:0/1717549957 shutdown_connections 2023-07-14T17:44:02.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.997+0000 7ff8f2658700 1 --2- 172.21.15.88:0/1717549957 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff8ec100110 0x7ff8ec19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.997+0000 7ff8f2658700 1 --2- 172.21.15.88:0/1717549957 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff8ec0fee00 0x7ff8ec195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.997+0000 7ff8f2658700 1 --2- 172.21.15.88:0/1717549957 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff8d40a11e0 0x7ff8d40a3690 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.997+0000 7ff8f2658700 1 --2- 172.21.15.88:0/1717549957 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8ec0ff7b0 0x7ff8ec195ed0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:02.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.997+0000 7ff8f2658700 1 -- 172.21.15.88:0/1717549957 >> 172.21.15.88:0/1717549957 conn(0x7ff8ec0f98a0 msgr2=0x7ff8ec0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:02.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.997+0000 7ff8f2658700 1 -- 172.21.15.88:0/1717549957 shutdown_connections 2023-07-14T17:44:02.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:02.997+0000 7ff8f2658700 1 -- 172.21.15.88:0/1717549957 wait complete. 2023-07-14T17:44:02.991 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T17:44:03.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.134+0000 7fd850b91700 1 -- 172.21.15.88:0/2817460827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd84c0fe620 msgr2=0x7fd84c1081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:03.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.134+0000 7fd850b91700 1 --2- 172.21.15.88:0/2817460827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd84c0fe620 0x7fd84c1081e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd84400b0a0 tx=0x7fd84400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.134+0000 7fd850b91700 1 -- 172.21.15.88:0/2817460827 shutdown_connections 2023-07-14T17:44:03.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.134+0000 7fd850b91700 1 --2- 172.21.15.88:0/2817460827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd84c0fe620 0x7fd84c1081e0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.134+0000 7fd850b91700 1 --2- 172.21.15.88:0/2817460827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c101200 0x7fd84c101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.134+0000 7fd850b91700 1 --2- 172.21.15.88:0/2817460827 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd84c0fdc70 0x7fd84c0fe0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.134+0000 7fd850b91700 1 -- 172.21.15.88:0/2817460827 >> 172.21.15.88:0/2817460827 conn(0x7fd84c0f98a0 msgr2=0x7fd84c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:03.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.134+0000 7fd850b91700 1 -- 172.21.15.88:0/2817460827 shutdown_connections 2023-07-14T17:44:03.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.134+0000 7fd850b91700 1 -- 172.21.15.88:0/2817460827 wait complete. 2023-07-14T17:44:03.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.134+0000 7fd850b91700 1 Processor -- start 2023-07-14T17:44:03.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.134+0000 7fd850b91700 1 -- start start 2023-07-14T17:44:03.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd850b91700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c0fdc70 0x7fd84c195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:03.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd850b91700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd84c0fe620 0x7fd84c195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:03.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd850b91700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd84c101200 0x7fd84c19cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:03.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd850b91700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c19d510 con 0x7fd84c0fe620 2023-07-14T17:44:03.119 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd850b91700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c19d650 con 0x7fd84c0fdc70 2023-07-14T17:44:03.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd850b91700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84c19d7e0 con 0x7fd84c101200 2023-07-14T17:44:03.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd849d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd84c0fe620 0x7fd84c195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:03.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd849d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd84c0fe620 0x7fd84c195ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42572/0 (socket says 172.21.15.88:42572) 2023-07-14T17:44:03.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd849d9b700 1 -- 172.21.15.88:0/813349269 learned_addr learned my addr 172.21.15.88:0/813349269 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:03.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd84ad9d700 1 --2- 172.21.15.88:0/813349269 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd84c101200 0x7fd84c19cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:03.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd84a59c700 1 --2- 172.21.15.88:0/813349269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c0fdc70 0x7fd84c195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:03.120 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd849d9b700 1 -- 172.21.15.88:0/813349269 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd84c101200 msgr2=0x7fd84c19cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:03.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd849d9b700 1 --2- 172.21.15.88:0/813349269 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd84c101200 0x7fd84c19cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd849d9b700 1 -- 172.21.15.88:0/813349269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c0fdc70 msgr2=0x7fd84c195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:03.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd849d9b700 1 --2- 172.21.15.88:0/813349269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c0fdc70 0x7fd84c195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd849d9b700 1 -- 172.21.15.88:0/813349269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd844009d00 con 0x7fd84c0fe620 2023-07-14T17:44:03.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd849d9b700 1 --2- 172.21.15.88:0/813349269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd84c0fe620 0x7fd84c195ed0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd83c00d980 tx=0x7fd83c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:03.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd83b7fe700 1 -- 172.21.15.88:0/813349269 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c004020 con 0x7fd84c0fe620 2023-07-14T17:44:03.121 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd83b7fe700 1 -- 172.21.15.88:0/813349269 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd83c00f690 con 0x7fd84c0fe620 2023-07-14T17:44:03.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd850b91700 1 -- 172.21.15.88:0/813349269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd84c19dac0 con 0x7fd84c0fe620 2023-07-14T17:44:03.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd83b7fe700 1 -- 172.21.15.88:0/813349269 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83c0039e0 con 0x7fd84c0fe620 2023-07-14T17:44:03.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.135+0000 7fd850b91700 1 -- 172.21.15.88:0/813349269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd84c19dfd0 con 0x7fd84c0fe620 2023-07-14T17:44:03.122 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.137+0000 7fd850b91700 1 -- 172.21.15.88:0/813349269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd84c106d20 con 0x7fd84c0fe620 2023-07-14T17:44:03.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.138+0000 7fd83b7fe700 1 -- 172.21.15.88:0/813349269 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd83c00f800 con 0x7fd84c0fe620 2023-07-14T17:44:03.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.138+0000 7fd83b7fe700 1 --2- 172.21.15.88:0/813349269 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd8340a11e0 0x7fd8340a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:03.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.139+0000 7fd84a59c700 1 --2- 172.21.15.88:0/813349269 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd8340a11e0 0x7fd8340a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:03.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.139+0000 7fd83b7fe700 1 -- 172.21.15.88:0/813349269 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd83c0cccf0 con 0x7fd84c0fe620 2023-07-14T17:44:03.123 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.139+0000 7fd84a59c700 1 --2- 172.21.15.88:0/813349269 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd8340a11e0 0x7fd8340a3690 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd84c0674a0 tx=0x7fd84000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:03.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.142+0000 7fd83b7fe700 1 -- 172.21.15.88:0/813349269 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd83c095fd0 con 0x7fd84c0fe620 2023-07-14T17:44:03.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:02 smithi107 ceph-mon[128999]: pgmap v297: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2023-07-14T17:44:03.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:02 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3680579941' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:03.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:02 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3751378170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:44:03.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:02 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1499237755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:44:03.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.332+0000 7fd850b91700 1 -- 172.21.15.88:0/813349269 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd84c02ca70 con 0x7fd84c0fe620 2023-07-14T17:44:03.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.332+0000 7fd83b7fe700 1 -- 172.21.15.88:0/813349269 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd83c0995f0 con 0x7fd84c0fe620 2023-07-14T17:44:03.315 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-07-14T17:44:03.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.335+0000 7fd850b91700 1 -- 172.21.15.88:0/813349269 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd8340a11e0 msgr2=0x7fd8340a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:03.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.335+0000 7fd850b91700 1 --2- 172.21.15.88:0/813349269 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd8340a11e0 0x7fd8340a3690 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd84c0674a0 tx=0x7fd84000b410 comp rx=0 tx=0).stop 2023-07-14T17:44:03.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.335+0000 7fd850b91700 1 -- 172.21.15.88:0/813349269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd84c0fe620 msgr2=0x7fd84c195ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:03.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.335+0000 7fd850b91700 1 --2- 172.21.15.88:0/813349269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd84c0fe620 0x7fd84c195ed0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd83c00d980 tx=0x7fd83c00dc90 comp rx=0 tx=0).stop 2023-07-14T17:44:03.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.335+0000 7fd850b91700 1 -- 172.21.15.88:0/813349269 shutdown_connections 2023-07-14T17:44:03.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.335+0000 7fd850b91700 1 --2- 172.21.15.88:0/813349269 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd84c101200 0x7fd84c19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.335+0000 7fd850b91700 1 --2- 172.21.15.88:0/813349269 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd84c0fdc70 0x7fd84c195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.335+0000 7fd850b91700 1 --2- 172.21.15.88:0/813349269 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd8340a11e0 0x7fd8340a3690 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.335+0000 7fd850b91700 1 --2- 172.21.15.88:0/813349269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd84c0fe620 0x7fd84c195ed0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.335+0000 7fd850b91700 1 -- 172.21.15.88:0/813349269 >> 172.21.15.88:0/813349269 conn(0x7fd84c0f98a0 msgr2=0x7fd84c0fbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:03.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.335+0000 7fd850b91700 1 -- 172.21.15.88:0/813349269 shutdown_connections 2023-07-14T17:44:03.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.335+0000 7fd850b91700 1 -- 172.21.15.88:0/813349269 wait complete. 2023-07-14T17:44:03.329 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:44:03.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:02 smithi178 ceph-mon[129368]: pgmap v297: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2023-07-14T17:44:03.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:02 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3680579941' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:03.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:02 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3751378170' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:44:03.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:02 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1499237755' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:44:03.449 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.467+0000 7f107be88700 1 -- 172.21.15.88:0/614515535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1074103040 msgr2=0x7f1074105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:03.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.467+0000 7f107be88700 1 --2- 172.21.15.88:0/614515535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1074103040 0x7f1074105420 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1070009990 tx=0x7f1070009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.467+0000 7f107be88700 1 -- 172.21.15.88:0/614515535 shutdown_connections 2023-07-14T17:44:03.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.467+0000 7f107be88700 1 --2- 172.21.15.88:0/614515535 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1074105960 0x7f1074107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.467+0000 7f107be88700 1 --2- 172.21.15.88:0/614515535 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1074100720 0x7f1074102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.467+0000 7f107be88700 1 --2- 172.21.15.88:0/614515535 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1074103040 0x7f1074105420 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.450 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.467+0000 7f107be88700 1 -- 172.21.15.88:0/614515535 >> 172.21.15.88:0/614515535 conn(0x7f10740fa100 msgr2=0x7f10740fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:03.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.468+0000 7f107be88700 1 -- 172.21.15.88:0/614515535 shutdown_connections 2023-07-14T17:44:03.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.468+0000 7f107be88700 1 -- 172.21.15.88:0/614515535 wait complete. 2023-07-14T17:44:03.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.468+0000 7f107be88700 1 Processor -- start 2023-07-14T17:44:03.451 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.468+0000 7f107be88700 1 -- start start 2023-07-14T17:44:03.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.468+0000 7f107be88700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1074100720 0x7f1074195840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:03.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.468+0000 7f107be88700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1074103040 0x7f1074195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:03.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.468+0000 7f107be88700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1074105960 0x7f107419cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:03.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.468+0000 7f107be88700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107419d3c0 con 0x7f1074103040 2023-07-14T17:44:03.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.468+0000 7f107be88700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107419d500 con 0x7f1074105960 2023-07-14T17:44:03.453 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.468+0000 7f107be88700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107419d690 con 0x7f1074100720 2023-07-14T17:44:03.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.468+0000 7f1079423700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1074103040 0x7f1074195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:03.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.468+0000 7f1079423700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1074103040 0x7f1074195d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42586/0 (socket says 172.21.15.88:42586) 2023-07-14T17:44:03.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.468+0000 7f1079423700 1 -- 172.21.15.88:0/2280462055 learned_addr learned my addr 172.21.15.88:0/2280462055 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:03.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.468+0000 7f107a425700 1 --2- 172.21.15.88:0/2280462055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1074105960 0x7f107419cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:03.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.469+0000 7f1079c24700 1 --2- 172.21.15.88:0/2280462055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1074100720 0x7f1074195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:03.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.469+0000 7f1079423700 1 -- 172.21.15.88:0/2280462055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1074100720 msgr2=0x7f1074195840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:03.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.469+0000 7f1079423700 1 --2- 172.21.15.88:0/2280462055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1074100720 0x7f1074195840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.469+0000 7f1079423700 1 -- 172.21.15.88:0/2280462055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1074105960 msgr2=0x7f107419cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:03.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.469+0000 7f1079423700 1 --2- 172.21.15.88:0/2280462055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1074105960 0x7f107419cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.469+0000 7f1079423700 1 -- 172.21.15.88:0/2280462055 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10700095f0 con 0x7f1074103040 2023-07-14T17:44:03.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.469+0000 7f1079c24700 1 --2- 172.21.15.88:0/2280462055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1074100720 0x7f1074195840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:03.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.469+0000 7f1079423700 1 --2- 172.21.15.88:0/2280462055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1074103040 0x7f1074195d80 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1070005950 tx=0x7f1070009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:03.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.469+0000 7f106affd700 1 -- 172.21.15.88:0/2280462055 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107001d070 con 0x7f1074103040 2023-07-14T17:44:03.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.469+0000 7f106affd700 1 -- 172.21.15.88:0/2280462055 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f107000bd70 con 0x7f1074103040 2023-07-14T17:44:03.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.469+0000 7f107be88700 1 -- 172.21.15.88:0/2280462055 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107419d910 con 0x7f1074103040 2023-07-14T17:44:03.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.469+0000 7f106affd700 1 -- 172.21.15.88:0/2280462055 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1070022d40 con 0x7f1074103040 2023-07-14T17:44:03.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.469+0000 7f107be88700 1 -- 172.21.15.88:0/2280462055 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107419dd90 con 0x7f1074103040 2023-07-14T17:44:03.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.470+0000 7f107be88700 1 -- 172.21.15.88:0/2280462055 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107410b2b0 con 0x7f1074103040 2023-07-14T17:44:03.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.471+0000 7f106affd700 1 -- 172.21.15.88:0/2280462055 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f1070022470 con 0x7f1074103040 2023-07-14T17:44:03.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.472+0000 7f106affd700 1 --2- 172.21.15.88:0/2280462055 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f10600a12a0 0x7f10600a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:03.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.472+0000 7f1079c24700 1 --2- 172.21.15.88:0/2280462055 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f10600a12a0 0x7f10600a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:03.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.472+0000 7f106affd700 1 -- 172.21.15.88:0/2280462055 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f10700ce9c0 con 0x7f1074103040 2023-07-14T17:44:03.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.472+0000 7f1079c24700 1 --2- 172.21.15.88:0/2280462055 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f10600a12a0 0x7f10600a3750 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1064005950 tx=0x7f10640058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:03.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.475+0000 7f106affd700 1 -- 172.21.15.88:0/2280462055 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1070097d30 con 0x7f1074103040 2023-07-14T17:44:03.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.707+0000 7f107be88700 1 -- 172.21.15.88:0/2280462055 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f107419e3d0 con 0x7f1074103040 2023-07-14T17:44:03.689 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.707+0000 7f106affd700 1 -- 172.21.15.88:0/2280462055 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10885 (secure 0 0 0) 0x7f1070005c00 con 0x7f1074103040 2023-07-14T17:44:03.689 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:03.689 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T17:42:42.314442+0000","modified":"2023-07-14T17:43:04.774381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24473,"mds_2":24469,"mds_3":24454,"mds_4":24449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6839/3046592155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3046592155},{"type":"v1","addr":"172.21.15.88:6839","nonce":3046592155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6841/2509502918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6840","nonce":2509502918},{"type":"v1","addr":"172.21.15.88:6841","nonce":2509502918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6835/2332025068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":2332025068},{"type":"v1","addr":"172.21.15.178:6835","nonce":2332025068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24449":{"gid":24449,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.107:6835/3062637672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3062637672},{"type":"v1","addr":"172.21.15.107:6835","nonce":3062637672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.178:6837/3950404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3950404084},{"type":"v1","addr":"172.21.15.178:6837","nonce":3950404084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.178:6839/2953415077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2953415077},{"type":"v1","addr":"172.21.15.178:6839","nonce":2953415077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/1152608615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1152608615},{"type":"v1","addr":"172.21.15.107:6837","nonce":1152608615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6839/31646548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":31646548},{"type":"v1","addr":"172.21.15.107:6839","nonce":31646548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T17:44:03.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.709+0000 7f107be88700 1 -- 172.21.15.88:0/2280462055 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f10600a12a0 msgr2=0x7f10600a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:03.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.709+0000 7f107be88700 1 --2- 172.21.15.88:0/2280462055 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f10600a12a0 0x7f10600a3750 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1064005950 tx=0x7f10640058e0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.709+0000 7f107be88700 1 -- 172.21.15.88:0/2280462055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1074103040 msgr2=0x7f1074195d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:03.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.709+0000 7f107be88700 1 --2- 172.21.15.88:0/2280462055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1074103040 0x7f1074195d80 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1070005950 tx=0x7f1070009580 comp rx=0 tx=0).stop 2023-07-14T17:44:03.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.709+0000 7f107be88700 1 -- 172.21.15.88:0/2280462055 shutdown_connections 2023-07-14T17:44:03.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.709+0000 7f107be88700 1 --2- 172.21.15.88:0/2280462055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1074105960 0x7f107419cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.709+0000 7f107be88700 1 --2- 172.21.15.88:0/2280462055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1074100720 0x7f1074195840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.709+0000 7f107be88700 1 --2- 172.21.15.88:0/2280462055 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f10600a12a0 0x7f10600a3750 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.709+0000 7f107be88700 1 --2- 172.21.15.88:0/2280462055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1074103040 0x7f1074195d80 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.709+0000 7f107be88700 1 -- 172.21.15.88:0/2280462055 >> 172.21.15.88:0/2280462055 conn(0x7f10740fa100 msgr2=0x7f1074101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:03.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.710+0000 7f107be88700 1 -- 172.21.15.88:0/2280462055 shutdown_connections 2023-07-14T17:44:03.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.710+0000 7f107be88700 1 -- 172.21.15.88:0/2280462055 wait complete. 2023-07-14T17:44:03.693 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 27 2023-07-14T17:44:03.704 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:44:03.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.844+0000 7f11a24f9700 1 -- 172.21.15.88:0/1506787882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f119c0ffcf0 msgr2=0x7f119c100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:03.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.844+0000 7f11a24f9700 1 --2- 172.21.15.88:0/1506787882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f119c0ffcf0 0x7f119c100140 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1184009990 tx=0x7f1184009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.844+0000 7f11a24f9700 1 -- 172.21.15.88:0/1506787882 shutdown_connections 2023-07-14T17:44:03.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.844+0000 7f11a24f9700 1 --2- 172.21.15.88:0/1506787882 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f119c100680 0x7f119c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.844+0000 7f11a24f9700 1 --2- 172.21.15.88:0/1506787882 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f119c0feb00 0x7f119c0fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.844+0000 7f11a24f9700 1 --2- 172.21.15.88:0/1506787882 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f119c0ffcf0 0x7f119c100140 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.844+0000 7f11a24f9700 1 -- 172.21.15.88:0/1506787882 >> 172.21.15.88:0/1506787882 conn(0x7f119c0fa100 msgr2=0x7f119c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:03.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.844+0000 7f11a24f9700 1 -- 172.21.15.88:0/1506787882 shutdown_connections 2023-07-14T17:44:03.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.844+0000 7f11a24f9700 1 -- 172.21.15.88:0/1506787882 wait complete. 2023-07-14T17:44:03.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f11a24f9700 1 Processor -- start 2023-07-14T17:44:03.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f11a24f9700 1 -- start start 2023-07-14T17:44:03.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f11a24f9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f119c0feb00 0x7f119c068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:03.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f11a24f9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f119c0ffcf0 0x7f119c19a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:03.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f11a24f9700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f119c100680 0x7f119c19cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:03.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f11a24f9700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119c19d360 con 0x7f119c0ffcf0 2023-07-14T17:44:03.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f11a24f9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119c19d4d0 con 0x7f119c0feb00 2023-07-14T17:44:03.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f11a24f9700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f119c19d690 con 0x7f119c100680 2023-07-14T17:44:03.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f1193fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f119c0ffcf0 0x7f119c19a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:03.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f1193fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f119c0ffcf0 0x7f119c19a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42600/0 (socket says 172.21.15.88:42600) 2023-07-14T17:44:03.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f1193fff700 1 -- 172.21.15.88:0/3273945713 learned_addr learned my addr 172.21.15.88:0/3273945713 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:03.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f119bfff700 1 --2- 172.21.15.88:0/3273945713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f119c0feb00 0x7f119c068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:03.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f11a0a96700 1 --2- 172.21.15.88:0/3273945713 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f119c100680 0x7f119c19cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:03.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f1193fff700 1 -- 172.21.15.88:0/3273945713 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f119c100680 msgr2=0x7f119c19cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:03.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f1193fff700 1 --2- 172.21.15.88:0/3273945713 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f119c100680 0x7f119c19cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f1193fff700 1 -- 172.21.15.88:0/3273945713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f119c0feb00 msgr2=0x7f119c068330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:03.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f1193fff700 1 --2- 172.21.15.88:0/3273945713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f119c0feb00 0x7f119c068330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:03.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f1193fff700 1 -- 172.21.15.88:0/3273945713 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11840095f0 con 0x7f119c0ffcf0 2023-07-14T17:44:03.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f11a0a96700 1 --2- 172.21.15.88:0/3273945713 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f119c100680 0x7f119c19cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:03.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.845+0000 7f1193fff700 1 --2- 172.21.15.88:0/3273945713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f119c0ffcf0 0x7f119c19a540 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1184003a50 tx=0x7f1184009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:03.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.846+0000 7f1199ffb700 1 -- 172.21.15.88:0/3273945713 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118401d070 con 0x7f119c0ffcf0 2023-07-14T17:44:03.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.846+0000 7f1199ffb700 1 -- 172.21.15.88:0/3273945713 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1184005a00 con 0x7f119c0ffcf0 2023-07-14T17:44:03.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.846+0000 7f11a24f9700 1 -- 172.21.15.88:0/3273945713 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f119c19d910 con 0x7f119c0ffcf0 2023-07-14T17:44:03.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.846+0000 7f1199ffb700 1 -- 172.21.15.88:0/3273945713 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1184005020 con 0x7f119c0ffcf0 2023-07-14T17:44:03.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.846+0000 7f11a24f9700 1 -- 172.21.15.88:0/3273945713 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f119c19dd60 con 0x7f119c0ffcf0 2023-07-14T17:44:03.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.847+0000 7f11a24f9700 1 -- 172.21.15.88:0/3273945713 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f119c10a180 con 0x7f119c0ffcf0 2023-07-14T17:44:03.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.848+0000 7f1199ffb700 1 -- 172.21.15.88:0/3273945713 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f11840051c0 con 0x7f119c0ffcf0 2023-07-14T17:44:03.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.848+0000 7f1199ffb700 1 --2- 172.21.15.88:0/3273945713 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f11880a1220 0x7f11880a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:03.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.848+0000 7f119bfff700 1 --2- 172.21.15.88:0/3273945713 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f11880a1220 0x7f11880a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:03.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.848+0000 7f1199ffb700 1 -- 172.21.15.88:0/3273945713 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f11840cdf90 con 0x7f119c0ffcf0 2023-07-14T17:44:03.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.849+0000 7f119bfff700 1 --2- 172.21.15.88:0/3273945713 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f11880a1220 0x7f11880a36d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f119c0ffb50 tx=0x7f118c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:03.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:03.853+0000 7f1199ffb700 1 -- 172.21.15.88:0/3273945713 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1184097420 con 0x7f119c0ffcf0 2023-07-14T17:44:04.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.049+0000 7f11a24f9700 1 -- 172.21.15.88:0/3273945713 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f119c105a50 con 0x7f119c0ffcf0 2023-07-14T17:44:04.032 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.050+0000 7f1199ffb700 1 -- 172.21.15.88:0/3273945713 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24512 (secure 0 0 0) 0x7f118409aa40 con 0x7f119c0ffcf0 2023-07-14T17:44:04.032 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:04.032 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":88,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:43:44.887959+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:44:04.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.052+0000 7f11a24f9700 1 -- 172.21.15.88:0/3273945713 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f11880a1220 msgr2=0x7f11880a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:04.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.052+0000 7f11a24f9700 1 --2- 172.21.15.88:0/3273945713 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f11880a1220 0x7f11880a36d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f119c0ffb50 tx=0x7f118c00b410 comp rx=0 tx=0).stop 2023-07-14T17:44:04.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.052+0000 7f11a24f9700 1 -- 172.21.15.88:0/3273945713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f119c0ffcf0 msgr2=0x7f119c19a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:04.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.052+0000 7f11a24f9700 1 --2- 172.21.15.88:0/3273945713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f119c0ffcf0 0x7f119c19a540 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1184003a50 tx=0x7f1184009580 comp rx=0 tx=0).stop 2023-07-14T17:44:04.034 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.052+0000 7f11a24f9700 1 -- 172.21.15.88:0/3273945713 shutdown_connections 2023-07-14T17:44:04.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.052+0000 7f11a24f9700 1 --2- 172.21.15.88:0/3273945713 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f119c100680 0x7f119c19cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.052+0000 7f11a24f9700 1 --2- 172.21.15.88:0/3273945713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f119c0feb00 0x7f119c068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.052+0000 7f11a24f9700 1 --2- 172.21.15.88:0/3273945713 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f11880a1220 0x7f11880a36d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.052+0000 7f11a24f9700 1 --2- 172.21.15.88:0/3273945713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f119c0ffcf0 0x7f119c19a540 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.052+0000 7f11a24f9700 1 -- 172.21.15.88:0/3273945713 >> 172.21.15.88:0/3273945713 conn(0x7f119c0fa100 msgr2=0x7f119c0fc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:04.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.052+0000 7f11a24f9700 1 -- 172.21.15.88:0/3273945713 shutdown_connections 2023-07-14T17:44:04.035 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.052+0000 7f11a24f9700 1 -- 172.21.15.88:0/3273945713 wait complete. 2023-07-14T17:44:04.045 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:03 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1717549957' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:44:04.046 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:03 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/813349269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:44:04.046 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:03 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2280462055' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:04.047 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T17:38:50.126535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T17:42:40.725870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T17:42:41.512749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T17:42:44.764573+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T17:44:04.048 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T17:44:04.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.190+0000 7f384d126700 1 -- 172.21.15.88:0/3252348827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3848103010 msgr2=0x7f38481053f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:04.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.190+0000 7f384d126700 1 --2- 172.21.15.88:0/3252348827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3848103010 0x7f38481053f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f3838009990 tx=0x7f3838009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.190+0000 7f384d126700 1 -- 172.21.15.88:0/3252348827 shutdown_connections 2023-07-14T17:44:04.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.190+0000 7f384d126700 1 --2- 172.21.15.88:0/3252348827 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3848105930 0x7f3848107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.190+0000 7f384d126700 1 --2- 172.21.15.88:0/3252348827 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38481006f0 0x7f3848102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.190+0000 7f384d126700 1 --2- 172.21.15.88:0/3252348827 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3848103010 0x7f38481053f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.190+0000 7f384d126700 1 -- 172.21.15.88:0/3252348827 >> 172.21.15.88:0/3252348827 conn(0x7f38480fa130 msgr2=0x7f38480fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:04.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.190+0000 7f384d126700 1 -- 172.21.15.88:0/3252348827 shutdown_connections 2023-07-14T17:44:04.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.190+0000 7f384d126700 1 -- 172.21.15.88:0/3252348827 wait complete. 2023-07-14T17:44:04.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.190+0000 7f384d126700 1 Processor -- start 2023-07-14T17:44:04.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f384d126700 1 -- start start 2023-07-14T17:44:04.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f384d126700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38481006f0 0x7f38481958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:04.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f384d126700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3848103010 0x7f3848195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:04.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f384d126700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3848105930 0x7f384819ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:04.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f384d126700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384819d420 con 0x7f3848105930 2023-07-14T17:44:04.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f384d126700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384819d560 con 0x7f3848103010 2023-07-14T17:44:04.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f384d126700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384819d6f0 con 0x7f38481006f0 2023-07-14T17:44:04.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f384759e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3848105930 0x7f384819ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:04.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f384759e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3848105930 0x7f384819ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42616/0 (socket says 172.21.15.88:42616) 2023-07-14T17:44:04.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f384759e700 1 -- 172.21.15.88:0/57415403 learned_addr learned my addr 172.21.15.88:0/57415403 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:04.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f3846d9d700 1 --2- 172.21.15.88:0/57415403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38481006f0 0x7f38481958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:04.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f384659c700 1 --2- 172.21.15.88:0/57415403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3848103010 0x7f3848195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:04.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f384759e700 1 -- 172.21.15.88:0/57415403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38481006f0 msgr2=0x7f38481958a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:04.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f384759e700 1 --2- 172.21.15.88:0/57415403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38481006f0 0x7f38481958a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f384759e700 1 -- 172.21.15.88:0/57415403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3848103010 msgr2=0x7f3848195de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:04.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f384759e700 1 --2- 172.21.15.88:0/57415403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3848103010 0x7f3848195de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f384759e700 1 -- 172.21.15.88:0/57415403 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38380095f0 con 0x7f3848105930 2023-07-14T17:44:04.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f384759e700 1 --2- 172.21.15.88:0/57415403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3848105930 0x7f384819ce50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3840007ee0 tx=0x7f384000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:04.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f382ffff700 1 -- 172.21.15.88:0/57415403 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3840017070 con 0x7f3848105930 2023-07-14T17:44:04.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f384d126700 1 -- 172.21.15.88:0/57415403 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f384819d9d0 con 0x7f3848105930 2023-07-14T17:44:04.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f382ffff700 1 -- 172.21.15.88:0/57415403 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f384000f040 con 0x7f3848105930 2023-07-14T17:44:04.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.191+0000 7f382ffff700 1 -- 172.21.15.88:0/57415403 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3840005710 con 0x7f3848105930 2023-07-14T17:44:04.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.192+0000 7f384d126700 1 -- 172.21.15.88:0/57415403 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f384819de90 con 0x7f3848105930 2023-07-14T17:44:04.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.192+0000 7f384d126700 1 -- 172.21.15.88:0/57415403 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38480fbca0 con 0x7f3848105930 2023-07-14T17:44:04.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.193+0000 7f382ffff700 1 -- 172.21.15.88:0/57415403 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3840005870 con 0x7f3848105930 2023-07-14T17:44:04.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.194+0000 7f382ffff700 1 --2- 172.21.15.88:0/57415403 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f38300a11e0 0x7f38300a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:04.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.194+0000 7f3846d9d700 1 --2- 172.21.15.88:0/57415403 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f38300a11e0 0x7f38300a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:04.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.194+0000 7f382ffff700 1 -- 172.21.15.88:0/57415403 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f38400cc5c0 con 0x7f3848105930 2023-07-14T17:44:04.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.194+0000 7f3846d9d700 1 --2- 172.21.15.88:0/57415403 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f38300a11e0 0x7f38300a3690 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f383c005950 tx=0x7f383c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:04.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.198+0000 7f382ffff700 1 -- 172.21.15.88:0/57415403 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f38400958a0 con 0x7f3848105930 2023-07-14T17:44:04.223 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:03 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1717549957' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:44:04.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:03 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/813349269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:44:04.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:03 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2280462055' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:04.367 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.386+0000 7f384d126700 1 -- 172.21.15.88:0/57415403 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f384802cb10 con 0x7f3848105930 2023-07-14T17:44:04.368 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.386+0000 7f382ffff700 1 -- 172.21.15.88:0/57415403 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3840095020 con 0x7f3848105930 2023-07-14T17:44:04.368 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-07-14T17:44:04.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.388+0000 7f384d126700 1 -- 172.21.15.88:0/57415403 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f38300a11e0 msgr2=0x7f38300a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:04.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.388+0000 7f384d126700 1 --2- 172.21.15.88:0/57415403 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f38300a11e0 0x7f38300a3690 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f383c005950 tx=0x7f383c00a300 comp rx=0 tx=0).stop 2023-07-14T17:44:04.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.388+0000 7f384d126700 1 -- 172.21.15.88:0/57415403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3848105930 msgr2=0x7f384819ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:04.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.388+0000 7f384d126700 1 --2- 172.21.15.88:0/57415403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3848105930 0x7f384819ce50 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3840007ee0 tx=0x7f384000d3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.388+0000 7f384d126700 1 -- 172.21.15.88:0/57415403 shutdown_connections 2023-07-14T17:44:04.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.388+0000 7f384d126700 1 --2- 172.21.15.88:0/57415403 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f38300a11e0 0x7f38300a3690 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.388+0000 7f384d126700 1 --2- 172.21.15.88:0/57415403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3848105930 0x7f384819ce50 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.388+0000 7f384d126700 1 --2- 172.21.15.88:0/57415403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f38481006f0 0x7f38481958a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.388+0000 7f384d126700 1 --2- 172.21.15.88:0/57415403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3848103010 0x7f3848195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.388+0000 7f384d126700 1 -- 172.21.15.88:0/57415403 >> 172.21.15.88:0/57415403 conn(0x7f38480fa130 msgr2=0x7f3848101350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.389+0000 7f384d126700 1 -- 172.21.15.88:0/57415403 shutdown_connections 2023-07-14T17:44:04.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.389+0000 7f384d126700 1 -- 172.21.15.88:0/57415403 wait complete. 2023-07-14T17:44:04.382 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T17:44:04.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:03 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1717549957' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:44:04.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:03 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/813349269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:44:04.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:03 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2280462055' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:04.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.525+0000 7f2d83062700 1 -- 172.21.15.88:0/2449390997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d7c1001c0 msgr2=0x7f2d7c1005d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:04.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.525+0000 7f2d83062700 1 --2- 172.21.15.88:0/2449390997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d7c1001c0 0x7f2d7c1005d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2d70009990 tx=0x7f2d70009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.525+0000 7f2d83062700 1 -- 172.21.15.88:0/2449390997 shutdown_connections 2023-07-14T17:44:04.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.525+0000 7f2d83062700 1 --2- 172.21.15.88:0/2449390997 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d7c0ff730 0x7f2d7c106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.525+0000 7f2d83062700 1 --2- 172.21.15.88:0/2449390997 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d7c1001c0 0x7f2d7c1005d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.525+0000 7f2d83062700 1 --2- 172.21.15.88:0/2449390997 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d7c0fed80 0x7f2d7c0ff1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.525+0000 7f2d83062700 1 -- 172.21.15.88:0/2449390997 >> 172.21.15.88:0/2449390997 conn(0x7f2d7c0f98e0 msgr2=0x7f2d7c0fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:04.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.525+0000 7f2d83062700 1 -- 172.21.15.88:0/2449390997 shutdown_connections 2023-07-14T17:44:04.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.525+0000 7f2d83062700 1 -- 172.21.15.88:0/2449390997 wait complete. 2023-07-14T17:44:04.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.525+0000 7f2d83062700 1 Processor -- start 2023-07-14T17:44:04.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d83062700 1 -- start start 2023-07-14T17:44:04.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d83062700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d7c0fed80 0x7f2d7c102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d83062700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d7c0ff730 0x7f2d7c101170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d83062700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d7c1001c0 0x7f2d7c1016b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d83062700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c101e50 con 0x7f2d7c0fed80 2023-07-14T17:44:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d83062700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c101fc0 con 0x7f2d7c1001c0 2023-07-14T17:44:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d83062700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d7c19f8f0 con 0x7f2d7c0ff730 2023-07-14T17:44:04.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d80dfe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d7c0fed80 0x7f2d7c102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d80dfe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d7c0fed80 0x7f2d7c102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42626/0 (socket says 172.21.15.88:42626) 2023-07-14T17:44:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d80dfe700 1 -- 172.21.15.88:0/2438937634 learned_addr learned my addr 172.21.15.88:0/2438937634 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d815ff700 1 --2- 172.21.15.88:0/2438937634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d7c1001c0 0x7f2d7c1016b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d7bfff700 1 --2- 172.21.15.88:0/2438937634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d7c0ff730 0x7f2d7c101170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d80dfe700 1 -- 172.21.15.88:0/2438937634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d7c0ff730 msgr2=0x7f2d7c101170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d80dfe700 1 --2- 172.21.15.88:0/2438937634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d7c0ff730 0x7f2d7c101170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d80dfe700 1 -- 172.21.15.88:0/2438937634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d7c1001c0 msgr2=0x7f2d7c1016b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:04.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d80dfe700 1 --2- 172.21.15.88:0/2438937634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d7c1001c0 0x7f2d7c1016b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d80dfe700 1 -- 172.21.15.88:0/2438937634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d700095f0 con 0x7f2d7c0fed80 2023-07-14T17:44:04.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d80dfe700 1 --2- 172.21.15.88:0/2438937634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d7c0fed80 0x7f2d7c102b30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2d700059d0 tx=0x7f2d700040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:04.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d79ffb700 1 -- 172.21.15.88:0/2438937634 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7001d070 con 0x7f2d7c0fed80 2023-07-14T17:44:04.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d83062700 1 -- 172.21.15.88:0/2438937634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d7c19fb70 con 0x7f2d7c0fed80 2023-07-14T17:44:04.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d79ffb700 1 -- 172.21.15.88:0/2438937634 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2d700041d0 con 0x7f2d7c0fed80 2023-07-14T17:44:04.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.526+0000 7f2d79ffb700 1 -- 172.21.15.88:0/2438937634 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d70021410 con 0x7f2d7c0fed80 2023-07-14T17:44:04.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.527+0000 7f2d83062700 1 -- 172.21.15.88:0/2438937634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d7c1a0070 con 0x7f2d7c0fed80 2023-07-14T17:44:04.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.527+0000 7f2d83062700 1 -- 172.21.15.88:0/2438937634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d7c10a1c0 con 0x7f2d7c0fed80 2023-07-14T17:44:04.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.529+0000 7f2d79ffb700 1 -- 172.21.15.88:0/2438937634 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2d7000f460 con 0x7f2d7c0fed80 2023-07-14T17:44:04.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.529+0000 7f2d79ffb700 1 --2- 172.21.15.88:0/2438937634 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f2d6c0a11e0 0x7f2d6c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:04.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.529+0000 7f2d7bfff700 1 --2- 172.21.15.88:0/2438937634 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f2d6c0a11e0 0x7f2d6c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:04.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.529+0000 7f2d79ffb700 1 -- 172.21.15.88:0/2438937634 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2d700ce540 con 0x7f2d7c0fed80 2023-07-14T17:44:04.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.530+0000 7f2d7bfff700 1 --2- 172.21.15.88:0/2438937634 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f2d6c0a11e0 0x7f2d6c0a3690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2d68009b30 tx=0x7f2d68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:04.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.532+0000 7f2d79ffb700 1 -- 172.21.15.88:0/2438937634 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2d700978b0 con 0x7f2d7c0fed80 2023-07-14T17:44:04.702 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.720+0000 7f2d83062700 1 -- 172.21.15.88:0/2438937634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2d7c10a380 con 0x7f2d7c0fed80 2023-07-14T17:44:04.703 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.721+0000 7f2d79ffb700 1 -- 172.21.15.88:0/2438937634 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2d7c10a380 con 0x7f2d7c0fed80 2023-07-14T17:44:04.703 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-07-14T17:44:04.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.723+0000 7f2d83062700 1 -- 172.21.15.88:0/2438937634 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f2d6c0a11e0 msgr2=0x7f2d6c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:04.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.723+0000 7f2d83062700 1 --2- 172.21.15.88:0/2438937634 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f2d6c0a11e0 0x7f2d6c0a3690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2d68009b30 tx=0x7f2d68009260 comp rx=0 tx=0).stop 2023-07-14T17:44:04.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.723+0000 7f2d83062700 1 -- 172.21.15.88:0/2438937634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d7c0fed80 msgr2=0x7f2d7c102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:04.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.723+0000 7f2d83062700 1 --2- 172.21.15.88:0/2438937634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d7c0fed80 0x7f2d7c102b30 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f2d700059d0 tx=0x7f2d700040b0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.723+0000 7f2d83062700 1 -- 172.21.15.88:0/2438937634 shutdown_connections 2023-07-14T17:44:04.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.723+0000 7f2d83062700 1 --2- 172.21.15.88:0/2438937634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f2d7c1001c0 0x7f2d7c1016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.723+0000 7f2d83062700 1 --2- 172.21.15.88:0/2438937634 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f2d6c0a11e0 0x7f2d6c0a3690 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.723+0000 7f2d83062700 1 --2- 172.21.15.88:0/2438937634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2d7c0fed80 0x7f2d7c102b30 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.723+0000 7f2d83062700 1 --2- 172.21.15.88:0/2438937634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f2d7c0ff730 0x7f2d7c101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.723+0000 7f2d83062700 1 -- 172.21.15.88:0/2438937634 >> 172.21.15.88:0/2438937634 conn(0x7f2d7c0f98e0 msgr2=0x7f2d7c0fdd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:04.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.723+0000 7f2d83062700 1 -- 172.21.15.88:0/2438937634 shutdown_connections 2023-07-14T17:44:04.706 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.723+0000 7f2d83062700 1 -- 172.21.15.88:0/2438937634 wait complete. 2023-07-14T17:44:04.717 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T17:44:04.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.860+0000 7f36bccd8700 1 -- 172.21.15.88:0/4219767771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36b80fed40 msgr2=0x7f36b80ff1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:04.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.860+0000 7f36bccd8700 1 --2- 172.21.15.88:0/4219767771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36b80fed40 0x7f36b80ff1b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f36a8009990 tx=0x7f36a8009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.860+0000 7f36bccd8700 1 -- 172.21.15.88:0/4219767771 shutdown_connections 2023-07-14T17:44:04.842 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.860+0000 7f36bccd8700 1 --2- 172.21.15.88:0/4219767771 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36b80ff6f0 0x7f36b8106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.860+0000 7f36bccd8700 1 --2- 172.21.15.88:0/4219767771 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36b8100180 0x7f36b8100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.860+0000 7f36bccd8700 1 --2- 172.21.15.88:0/4219767771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36b80fed40 0x7f36b80ff1b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.860+0000 7f36bccd8700 1 -- 172.21.15.88:0/4219767771 >> 172.21.15.88:0/4219767771 conn(0x7f36b80f98a0 msgr2=0x7f36b80fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:04.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.860+0000 7f36bccd8700 1 -- 172.21.15.88:0/4219767771 shutdown_connections 2023-07-14T17:44:04.843 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.860+0000 7f36bccd8700 1 -- 172.21.15.88:0/4219767771 wait complete. 2023-07-14T17:44:04.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.861+0000 7f36bccd8700 1 Processor -- start 2023-07-14T17:44:04.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.861+0000 7f36bccd8700 1 -- start start 2023-07-14T17:44:04.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.861+0000 7f36bccd8700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36b80fed40 0x7f36b81094d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:04.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.861+0000 7f36bccd8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36b80ff6f0 0x7f36b8107b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:04.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.861+0000 7f36bccd8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36b8100180 0x7f36b8108050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:04.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.861+0000 7f36bccd8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b81087f0 con 0x7f36b8100180 2023-07-14T17:44:04.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.861+0000 7f36bccd8700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b8108960 con 0x7f36b80ff6f0 2023-07-14T17:44:04.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.861+0000 7f36bccd8700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36b819f910 con 0x7f36b80fed40 2023-07-14T17:44:04.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.861+0000 7f36b6d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36b8100180 0x7f36b8108050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:04.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.861+0000 7f36b6d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36b8100180 0x7f36b8108050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42644/0 (socket says 172.21.15.88:42644) 2023-07-14T17:44:04.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.861+0000 7f36b6d9d700 1 -- 172.21.15.88:0/242180564 learned_addr learned my addr 172.21.15.88:0/242180564 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:04.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.861+0000 7f36b5d9b700 1 --2- 172.21.15.88:0/242180564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36b80ff6f0 0x7f36b8107b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:04.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.861+0000 7f36b659c700 1 --2- 172.21.15.88:0/242180564 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36b80fed40 0x7f36b81094d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:04.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.861+0000 7f36b6d9d700 1 -- 172.21.15.88:0/242180564 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36b80fed40 msgr2=0x7f36b81094d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:04.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.861+0000 7f36b6d9d700 1 --2- 172.21.15.88:0/242180564 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36b80fed40 0x7f36b81094d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.861+0000 7f36b6d9d700 1 -- 172.21.15.88:0/242180564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36b80ff6f0 msgr2=0x7f36b8107b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:04.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.861+0000 7f36b6d9d700 1 --2- 172.21.15.88:0/242180564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36b80ff6f0 0x7f36b8107b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:04.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.862+0000 7f36b6d9d700 1 -- 172.21.15.88:0/242180564 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a80095f0 con 0x7f36b8100180 2023-07-14T17:44:04.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.862+0000 7f36b6d9d700 1 --2- 172.21.15.88:0/242180564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36b8100180 0x7f36b8108050 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f36b0007ee0 tx=0x7f36b000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:04.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.862+0000 7f36a77fe700 1 -- 172.21.15.88:0/242180564 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b000ddd0 con 0x7f36b8100180 2023-07-14T17:44:04.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.862+0000 7f36bccd8700 1 -- 172.21.15.88:0/242180564 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b819fbf0 con 0x7f36b8100180 2023-07-14T17:44:04.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.862+0000 7f36a77fe700 1 -- 172.21.15.88:0/242180564 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f36b000f040 con 0x7f36b8100180 2023-07-14T17:44:04.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.862+0000 7f36a77fe700 1 -- 172.21.15.88:0/242180564 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b00047d0 con 0x7f36b8100180 2023-07-14T17:44:04.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.862+0000 7f36bccd8700 1 -- 172.21.15.88:0/242180564 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b81a0100 con 0x7f36b8100180 2023-07-14T17:44:04.848 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.863+0000 7f36bccd8700 1 -- 172.21.15.88:0/242180564 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b8105ae0 con 0x7f36b8100180 2023-07-14T17:44:04.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.864+0000 7f36a77fe700 1 -- 172.21.15.88:0/242180564 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f36b0003a00 con 0x7f36b8100180 2023-07-14T17:44:04.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.864+0000 7f36a77fe700 1 --2- 172.21.15.88:0/242180564 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f36a00a11e0 0x7f36a00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:04.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.865+0000 7f36b659c700 1 --2- 172.21.15.88:0/242180564 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f36a00a11e0 0x7f36a00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:04.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.865+0000 7f36a77fe700 1 -- 172.21.15.88:0/242180564 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f36b00cc730 con 0x7f36b8100180 2023-07-14T17:44:04.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.865+0000 7f36b659c700 1 --2- 172.21.15.88:0/242180564 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f36a00a11e0 0x7f36a00a3690 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f36ac00dad0 tx=0x7f36ac00d4b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:04.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:04.869+0000 7f36a77fe700 1 -- 172.21.15.88:0/242180564 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f36b004e780 con 0x7f36b8100180 2023-07-14T17:44:05.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.059+0000 7f36bccd8700 1 -- 172.21.15.88:0/242180564 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f36b802ca70 con 0x7f36b8100180 2023-07-14T17:44:05.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.060+0000 7f36a77fe700 1 -- 172.21.15.88:0/242180564 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f36b0095a10 con 0x7f36b8100180 2023-07-14T17:44:05.042 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 64 2023-07-14T17:44:05.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.062+0000 7f36bccd8700 1 -- 172.21.15.88:0/242180564 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f36a00a11e0 msgr2=0x7f36a00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:05.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.062+0000 7f36bccd8700 1 --2- 172.21.15.88:0/242180564 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f36a00a11e0 0x7f36a00a3690 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f36ac00dad0 tx=0x7f36ac00d4b0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.062+0000 7f36bccd8700 1 -- 172.21.15.88:0/242180564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36b8100180 msgr2=0x7f36b8108050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:05.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.062+0000 7f36bccd8700 1 --2- 172.21.15.88:0/242180564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36b8100180 0x7f36b8108050 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f36b0007ee0 tx=0x7f36b000d3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.062+0000 7f36bccd8700 1 -- 172.21.15.88:0/242180564 shutdown_connections 2023-07-14T17:44:05.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.062+0000 7f36bccd8700 1 --2- 172.21.15.88:0/242180564 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f36a00a11e0 0x7f36a00a3690 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.062+0000 7f36bccd8700 1 --2- 172.21.15.88:0/242180564 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f36b8100180 0x7f36b8108050 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.062+0000 7f36bccd8700 1 --2- 172.21.15.88:0/242180564 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f36b80fed40 0x7f36b81094d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.062+0000 7f36bccd8700 1 --2- 172.21.15.88:0/242180564 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f36b80ff6f0 0x7f36b8107b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.062+0000 7f36bccd8700 1 -- 172.21.15.88:0/242180564 >> 172.21.15.88:0/242180564 conn(0x7f36b80f98a0 msgr2=0x7f36b80fbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:05.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.062+0000 7f36bccd8700 1 -- 172.21.15.88:0/242180564 shutdown_connections 2023-07-14T17:44:05.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.062+0000 7f36bccd8700 1 -- 172.21.15.88:0/242180564 wait complete. 2023-07-14T17:44:05.054 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:04 smithi088 ceph-mon[127465]: pgmap v298: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T17:44:05.055 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:04 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3273945713' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:05.055 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:04 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/57415403' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:44:05.055 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:04 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2438937634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:44:05.056 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T17:44:05.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.198+0000 7f37a1085700 1 -- 172.21.15.88:0/807932368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c0ff750 msgr2=0x7f379c107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:05.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.198+0000 7f37a1085700 1 --2- 172.21.15.88:0/807932368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c0ff750 0x7f379c107020 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f379400b0a0 tx=0x7f379400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.198+0000 7f37a1085700 1 -- 172.21.15.88:0/807932368 shutdown_connections 2023-07-14T17:44:05.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.198+0000 7f37a1085700 1 --2- 172.21.15.88:0/807932368 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c0ff750 0x7f379c107020 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.198+0000 7f37a1085700 1 --2- 172.21.15.88:0/807932368 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f379c100120 0x7f379c100530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.198+0000 7f37a1085700 1 --2- 172.21.15.88:0/807932368 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f379c0fedc0 0x7f379c0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.198+0000 7f37a1085700 1 -- 172.21.15.88:0/807932368 >> 172.21.15.88:0/807932368 conn(0x7f379c0f98e0 msgr2=0x7f379c0fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:05.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.198+0000 7f37a1085700 1 -- 172.21.15.88:0/807932368 shutdown_connections 2023-07-14T17:44:05.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.198+0000 7f37a1085700 1 -- 172.21.15.88:0/807932368 wait complete. 2023-07-14T17:44:05.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.199+0000 7f37a1085700 1 Processor -- start 2023-07-14T17:44:05.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.199+0000 7f37a1085700 1 -- start start 2023-07-14T17:44:05.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.199+0000 7f37a1085700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f379c0fedc0 0x7f379c194820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:05.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.199+0000 7f37a1085700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f379c0ff750 0x7f379c194d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:05.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.199+0000 7f37a1085700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c100120 0x7f379c19bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:05.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.199+0000 7f37a1085700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c19c3a0 con 0x7f379c100120 2023-07-14T17:44:05.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.199+0000 7f37a1085700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c19c4e0 con 0x7f379c0ff750 2023-07-14T17:44:05.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.199+0000 7f37a1085700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c19c670 con 0x7f379c0fedc0 2023-07-14T17:44:05.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.200+0000 7f379b59e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c100120 0x7f379c19bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:05.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.200+0000 7f379b59e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c100120 0x7f379c19bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42658/0 (socket says 172.21.15.88:42658) 2023-07-14T17:44:05.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.200+0000 7f379b59e700 1 -- 172.21.15.88:0/3700598604 learned_addr learned my addr 172.21.15.88:0/3700598604 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:05.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.200+0000 7f379ad9d700 1 --2- 172.21.15.88:0/3700598604 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f379c0fedc0 0x7f379c194820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:05.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.200+0000 7f379a59c700 1 --2- 172.21.15.88:0/3700598604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f379c0ff750 0x7f379c194d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:05.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.200+0000 7f379b59e700 1 -- 172.21.15.88:0/3700598604 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f379c0fedc0 msgr2=0x7f379c194820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:05.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.200+0000 7f379b59e700 1 --2- 172.21.15.88:0/3700598604 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f379c0fedc0 0x7f379c194820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.200+0000 7f379b59e700 1 -- 172.21.15.88:0/3700598604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f379c0ff750 msgr2=0x7f379c194d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:05.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.200+0000 7f379b59e700 1 --2- 172.21.15.88:0/3700598604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f379c0ff750 0x7f379c194d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.200+0000 7f379b59e700 1 -- 172.21.15.88:0/3700598604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3794009d00 con 0x7f379c100120 2023-07-14T17:44:05.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.200+0000 7f379b59e700 1 --2- 172.21.15.88:0/3700598604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c100120 0x7f379c19bdd0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f37940062a0 tx=0x7f3794004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:05.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.200+0000 7f3783fff700 1 -- 172.21.15.88:0/3700598604 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379400e040 con 0x7f379c100120 2023-07-14T17:44:05.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.200+0000 7f37a1085700 1 -- 172.21.15.88:0/3700598604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f379c19c8f0 con 0x7f379c100120 2023-07-14T17:44:05.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.200+0000 7f3783fff700 1 -- 172.21.15.88:0/3700598604 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f37940094c0 con 0x7f379c100120 2023-07-14T17:44:05.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.200+0000 7f3783fff700 1 -- 172.21.15.88:0/3700598604 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379401bc70 con 0x7f379c100120 2023-07-14T17:44:05.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.200+0000 7f37a1085700 1 -- 172.21.15.88:0/3700598604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f379c19cd70 con 0x7f379c100120 2023-07-14T17:44:05.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.201+0000 7f37a1085700 1 -- 172.21.15.88:0/3700598604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f379c106120 con 0x7f379c100120 2023-07-14T17:44:05.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.202+0000 7f3783fff700 1 -- 172.21.15.88:0/3700598604 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f379401d890 con 0x7f379c100120 2023-07-14T17:44:05.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.203+0000 7f3783fff700 1 --2- 172.21.15.88:0/3700598604 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f37840a11e0 0x7f37840a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:05.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.203+0000 7f3783fff700 1 -- 172.21.15.88:0/3700598604 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f37940cf580 con 0x7f379c100120 2023-07-14T17:44:05.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.203+0000 7f379ad9d700 1 --2- 172.21.15.88:0/3700598604 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f37840a11e0 0x7f37840a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:05.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.203+0000 7f379ad9d700 1 --2- 172.21.15.88:0/3700598604 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f37840a11e0 0x7f37840a3690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f379c068750 tx=0x7f379000bef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:05.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.206+0000 7f3783fff700 1 -- 172.21.15.88:0/3700598604 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f37940988f0 con 0x7f379c100120 2023-07-14T17:44:05.223 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:04 smithi107 ceph-mon[128999]: pgmap v298: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T17:44:05.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:04 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3273945713' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:05.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:04 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/57415403' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:44:05.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:04 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2438937634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:44:05.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.396+0000 7f37a1085700 1 -- 172.21.15.88:0/3700598604 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f379c02cb50 con 0x7f379c100120 2023-07-14T17:44:05.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.397+0000 7f3783fff700 1 -- 172.21.15.88:0/3700598604 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7f379409bf10 con 0x7f379c100120 2023-07-14T17:44:05.379 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-07-14T17:44:05.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.399+0000 7f37a1085700 1 -- 172.21.15.88:0/3700598604 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f37840a11e0 msgr2=0x7f37840a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:05.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.399+0000 7f37a1085700 1 --2- 172.21.15.88:0/3700598604 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f37840a11e0 0x7f37840a3690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f379c068750 tx=0x7f379000bef0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.399+0000 7f37a1085700 1 -- 172.21.15.88:0/3700598604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c100120 msgr2=0x7f379c19bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:05.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.399+0000 7f37a1085700 1 --2- 172.21.15.88:0/3700598604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c100120 0x7f379c19bdd0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f37940062a0 tx=0x7f3794004db0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.399+0000 7f37a1085700 1 -- 172.21.15.88:0/3700598604 shutdown_connections 2023-07-14T17:44:05.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.399+0000 7f37a1085700 1 --2- 172.21.15.88:0/3700598604 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f37840a11e0 0x7f37840a3690 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.399+0000 7f37a1085700 1 --2- 172.21.15.88:0/3700598604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f379c100120 0x7f379c19bdd0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.399+0000 7f37a1085700 1 --2- 172.21.15.88:0/3700598604 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f379c0fedc0 0x7f379c194820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.399+0000 7f37a1085700 1 --2- 172.21.15.88:0/3700598604 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f379c0ff750 0x7f379c194d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.399+0000 7f37a1085700 1 -- 172.21.15.88:0/3700598604 >> 172.21.15.88:0/3700598604 conn(0x7f379c0f98e0 msgr2=0x7f379c103260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:05.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.399+0000 7f37a1085700 1 -- 172.21.15.88:0/3700598604 shutdown_connections 2023-07-14T17:44:05.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.399+0000 7f37a1085700 1 -- 172.21.15.88:0/3700598604 wait complete. 2023-07-14T17:44:05.392 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:44:05.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:04 smithi178 ceph-mon[129368]: pgmap v298: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T17:44:05.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:04 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3273945713' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:05.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:04 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/57415403' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:44:05.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:04 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2438937634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:44:05.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.532+0000 7f5ce191a700 1 -- 172.21.15.88:0/780522075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc1017b0 msgr2=0x7f5cdc108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:05.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.532+0000 7f5ce191a700 1 --2- 172.21.15.88:0/780522075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc1017b0 0x7f5cdc108070 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400b0a0 tx=0x7f5cd400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.532+0000 7f5ce191a700 1 -- 172.21.15.88:0/780522075 shutdown_connections 2023-07-14T17:44:05.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.532+0000 7f5ce191a700 1 --2- 172.21.15.88:0/780522075 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc1017b0 0x7f5cdc108070 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.532+0000 7f5ce191a700 1 --2- 172.21.15.88:0/780522075 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cdc0ffc30 0x7f5cdc100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.532+0000 7f5ce191a700 1 --2- 172.21.15.88:0/780522075 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5cdc100e20 0x7f5cdc101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.532+0000 7f5ce191a700 1 -- 172.21.15.88:0/780522075 >> 172.21.15.88:0/780522075 conn(0x7f5cdc0fb210 msgr2=0x7f5cdc0fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:05.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.532+0000 7f5ce191a700 1 -- 172.21.15.88:0/780522075 shutdown_connections 2023-07-14T17:44:05.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.532+0000 7f5ce191a700 1 -- 172.21.15.88:0/780522075 wait complete. 2023-07-14T17:44:05.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.532+0000 7f5ce191a700 1 Processor -- start 2023-07-14T17:44:05.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5ce191a700 1 -- start start 2023-07-14T17:44:05.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5ce191a700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5cdc0ffc30 0x7f5cdc195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:05.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5ce191a700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cdc100e20 0x7f5cdc195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:05.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5ce191a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc1017b0 0x7f5cdc19cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:05.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5ce191a700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc19d490 con 0x7f5cdc1017b0 2023-07-14T17:44:05.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5ce191a700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc19d5d0 con 0x7f5cdc0ffc30 2023-07-14T17:44:05.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5ce191a700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc19d760 con 0x7f5cdc100e20 2023-07-14T17:44:05.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5cdb7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc1017b0 0x7f5cdc19cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:05.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5cdb7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc1017b0 0x7f5cdc19cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42672/0 (socket says 172.21.15.88:42672) 2023-07-14T17:44:05.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5cdb7fe700 1 -- 172.21.15.88:0/820846377 learned_addr learned my addr 172.21.15.88:0/820846377 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:05.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5cda7fc700 1 --2- 172.21.15.88:0/820846377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cdc100e20 0x7f5cdc195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:05.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5cdaffd700 1 --2- 172.21.15.88:0/820846377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5cdc0ffc30 0x7f5cdc195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:05.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5cdb7fe700 1 -- 172.21.15.88:0/820846377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cdc100e20 msgr2=0x7f5cdc195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:05.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5cdb7fe700 1 --2- 172.21.15.88:0/820846377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cdc100e20 0x7f5cdc195e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5cdb7fe700 1 -- 172.21.15.88:0/820846377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5cdc0ffc30 msgr2=0x7f5cdc195910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:05.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5cdb7fe700 1 --2- 172.21.15.88:0/820846377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5cdc0ffc30 0x7f5cdc195910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5cdb7fe700 1 -- 172.21.15.88:0/820846377 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd4009d00 con 0x7f5cdc1017b0 2023-07-14T17:44:05.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5cdaffd700 1 --2- 172.21.15.88:0/820846377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5cdc0ffc30 0x7f5cdc195910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:05.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.533+0000 7f5cdb7fe700 1 --2- 172.21.15.88:0/820846377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc1017b0 0x7f5cdc19cec0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400b830 tx=0x7f5cd4004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:05.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.534+0000 7f5ce0918700 1 -- 172.21.15.88:0/820846377 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd400e040 con 0x7f5cdc1017b0 2023-07-14T17:44:05.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.534+0000 7f5ce191a700 1 -- 172.21.15.88:0/820846377 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cdc19d9e0 con 0x7f5cdc1017b0 2023-07-14T17:44:05.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.534+0000 7f5ce0918700 1 -- 172.21.15.88:0/820846377 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5cd4009440 con 0x7f5cdc1017b0 2023-07-14T17:44:05.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.534+0000 7f5ce0918700 1 -- 172.21.15.88:0/820846377 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd401dbe0 con 0x7f5cdc1017b0 2023-07-14T17:44:05.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.534+0000 7f5ce191a700 1 -- 172.21.15.88:0/820846377 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cdc19de00 con 0x7f5cdc1017b0 2023-07-14T17:44:05.521 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.534+0000 7f5ce191a700 1 -- 172.21.15.88:0/820846377 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cdc10b330 con 0x7f5cdc1017b0 2023-07-14T17:44:05.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.536+0000 7f5ce0918700 1 -- 172.21.15.88:0/820846377 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5cd4005450 con 0x7f5cdc1017b0 2023-07-14T17:44:05.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.536+0000 7f5ce0918700 1 --2- 172.21.15.88:0/820846377 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5cc40a1220 0x7f5cc40a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:05.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.536+0000 7f5cdaffd700 1 --2- 172.21.15.88:0/820846377 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5cc40a1220 0x7f5cc40a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:05.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.536+0000 7f5ce0918700 1 -- 172.21.15.88:0/820846377 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5cd40ced60 con 0x7f5cdc1017b0 2023-07-14T17:44:05.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.537+0000 7f5cdaffd700 1 --2- 172.21.15.88:0/820846377 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5cc40a1220 0x7f5cc40a36d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0005fd0 tx=0x7f5cd0005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:05.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.540+0000 7f5ce0918700 1 -- 172.21.15.88:0/820846377 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5cd40980d0 con 0x7f5cdc1017b0 2023-07-14T17:44:05.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.775+0000 7f5ce191a700 1 -- 172.21.15.88:0/820846377 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cdc106a90 con 0x7f5cdc1017b0 2023-07-14T17:44:05.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.775+0000 7f5ce0918700 1 -- 172.21.15.88:0/820846377 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10885 (secure 0 0 0) 0x7f5cd409b6f0 con 0x7f5cdc1017b0 2023-07-14T17:44:05.757 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:05.757 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T17:42:42.314442+0000","modified":"2023-07-14T17:43:04.774381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24473,"mds_2":24469,"mds_3":24454,"mds_4":24449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6839/3046592155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3046592155},{"type":"v1","addr":"172.21.15.88:6839","nonce":3046592155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6841/2509502918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6840","nonce":2509502918},{"type":"v1","addr":"172.21.15.88:6841","nonce":2509502918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6835/2332025068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":2332025068},{"type":"v1","addr":"172.21.15.178:6835","nonce":2332025068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24449":{"gid":24449,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.107:6835/3062637672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3062637672},{"type":"v1","addr":"172.21.15.107:6835","nonce":3062637672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.178:6837/3950404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3950404084},{"type":"v1","addr":"172.21.15.178:6837","nonce":3950404084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.178:6839/2953415077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2953415077},{"type":"v1","addr":"172.21.15.178:6839","nonce":2953415077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/1152608615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1152608615},{"type":"v1","addr":"172.21.15.107:6837","nonce":1152608615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6839/31646548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":31646548},{"type":"v1","addr":"172.21.15.107:6839","nonce":31646548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T17:44:05.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.777+0000 7f5ce191a700 1 -- 172.21.15.88:0/820846377 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5cc40a1220 msgr2=0x7f5cc40a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:05.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.777+0000 7f5ce191a700 1 --2- 172.21.15.88:0/820846377 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5cc40a1220 0x7f5cc40a36d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0005fd0 tx=0x7f5cd0005f00 comp rx=0 tx=0).stop 2023-07-14T17:44:05.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.777+0000 7f5ce191a700 1 -- 172.21.15.88:0/820846377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc1017b0 msgr2=0x7f5cdc19cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:05.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.777+0000 7f5ce191a700 1 --2- 172.21.15.88:0/820846377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc1017b0 0x7f5cdc19cec0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400b830 tx=0x7f5cd4004d90 comp rx=0 tx=0).stop 2023-07-14T17:44:05.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.777+0000 7f5ce191a700 1 -- 172.21.15.88:0/820846377 shutdown_connections 2023-07-14T17:44:05.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.777+0000 7f5ce191a700 1 --2- 172.21.15.88:0/820846377 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5cc40a1220 0x7f5cc40a36d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.777+0000 7f5ce191a700 1 --2- 172.21.15.88:0/820846377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5cdc1017b0 0x7f5cdc19cec0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.777+0000 7f5ce191a700 1 --2- 172.21.15.88:0/820846377 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5cdc0ffc30 0x7f5cdc195910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.777+0000 7f5ce191a700 1 --2- 172.21.15.88:0/820846377 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5cdc100e20 0x7f5cdc195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.777+0000 7f5ce191a700 1 -- 172.21.15.88:0/820846377 >> 172.21.15.88:0/820846377 conn(0x7f5cdc0fb210 msgr2=0x7f5cdc0fd450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:05.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.778+0000 7f5ce191a700 1 -- 172.21.15.88:0/820846377 shutdown_connections 2023-07-14T17:44:05.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.778+0000 7f5ce191a700 1 -- 172.21.15.88:0/820846377 wait complete. 2023-07-14T17:44:05.761 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 27 2023-07-14T17:44:05.772 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:44:05.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.912+0000 7fa864601700 1 -- 172.21.15.88:0/1500687755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa85c0ff6d0 msgr2=0x7fa85c106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:05.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.912+0000 7fa864601700 1 --2- 172.21.15.88:0/1500687755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa85c0ff6d0 0x7fa85c106f80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa85400b0a0 tx=0x7fa85400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.912+0000 7fa864601700 1 -- 172.21.15.88:0/1500687755 shutdown_connections 2023-07-14T17:44:05.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.912+0000 7fa864601700 1 --2- 172.21.15.88:0/1500687755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa85c0ff6d0 0x7fa85c106f80 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.912+0000 7fa864601700 1 --2- 172.21.15.88:0/1500687755 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa85c1000d0 0x7fa85c1004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.912+0000 7fa864601700 1 --2- 172.21.15.88:0/1500687755 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa85c0fed20 0x7fa85c0ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.912+0000 7fa864601700 1 -- 172.21.15.88:0/1500687755 >> 172.21.15.88:0/1500687755 conn(0x7fa85c0f98a0 msgr2=0x7fa85c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:05.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.912+0000 7fa864601700 1 -- 172.21.15.88:0/1500687755 shutdown_connections 2023-07-14T17:44:05.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.912+0000 7fa864601700 1 -- 172.21.15.88:0/1500687755 wait complete. 2023-07-14T17:44:05.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.913+0000 7fa864601700 1 Processor -- start 2023-07-14T17:44:05.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.913+0000 7fa864601700 1 -- start start 2023-07-14T17:44:05.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.913+0000 7fa864601700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa85c0fed20 0x7fa85c102ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:05.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.913+0000 7fa864601700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa85c0ff6d0 0x7fa85c101110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:05.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.913+0000 7fa864601700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa85c1000d0 0x7fa85c101650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:05.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.913+0000 7fa864601700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c101df0 con 0x7fa85c0fed20 2023-07-14T17:44:05.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.913+0000 7fa864601700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c101f60 con 0x7fa85c1000d0 2023-07-14T17:44:05.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.913+0000 7fa864601700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c19f940 con 0x7fa85c0ff6d0 2023-07-14T17:44:05.898 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.913+0000 7fa86239d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa85c0fed20 0x7fa85c102ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:05.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.913+0000 7fa86239d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa85c0fed20 0x7fa85c102ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42682/0 (socket says 172.21.15.88:42682) 2023-07-14T17:44:05.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.913+0000 7fa86239d700 1 -- 172.21.15.88:0/685600365 learned_addr learned my addr 172.21.15.88:0/685600365 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:05.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.914+0000 7fa862b9e700 1 --2- 172.21.15.88:0/685600365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa85c1000d0 0x7fa85c101650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:05.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.914+0000 7fa86239d700 1 -- 172.21.15.88:0/685600365 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa85c0ff6d0 msgr2=0x7fa85c101110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:05.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.914+0000 7fa86239d700 1 --2- 172.21.15.88:0/685600365 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa85c0ff6d0 0x7fa85c101110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.914+0000 7fa86239d700 1 -- 172.21.15.88:0/685600365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa85c1000d0 msgr2=0x7fa85c101650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:05.899 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.914+0000 7fa86239d700 1 --2- 172.21.15.88:0/685600365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa85c1000d0 0x7fa85c101650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:05.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.914+0000 7fa86239d700 1 -- 172.21.15.88:0/685600365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa854009d00 con 0x7fa85c0fed20 2023-07-14T17:44:05.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.914+0000 7fa862b9e700 1 --2- 172.21.15.88:0/685600365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa85c1000d0 0x7fa85c101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:05.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.914+0000 7fa86239d700 1 --2- 172.21.15.88:0/685600365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa85c0fed20 0x7fa85c102ad0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa85800d980 tx=0x7fa85800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:05.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.914+0000 7fa8537fe700 1 -- 172.21.15.88:0/685600365 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85800bb60 con 0x7fa85c0fed20 2023-07-14T17:44:05.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.914+0000 7fa864601700 1 -- 172.21.15.88:0/685600365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa85c19fc20 con 0x7fa85c0fed20 2023-07-14T17:44:05.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.914+0000 7fa8537fe700 1 -- 172.21.15.88:0/685600365 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa858007620 con 0x7fa85c0fed20 2023-07-14T17:44:05.900 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.914+0000 7fa8537fe700 1 -- 172.21.15.88:0/685600365 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85800f3c0 con 0x7fa85c0fed20 2023-07-14T17:44:05.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.914+0000 7fa864601700 1 -- 172.21.15.88:0/685600365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa85c1a0130 con 0x7fa85c0fed20 2023-07-14T17:44:05.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.915+0000 7fa864601700 1 -- 172.21.15.88:0/685600365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa85c10a190 con 0x7fa85c0fed20 2023-07-14T17:44:05.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.916+0000 7fa8537fe700 1 -- 172.21.15.88:0/685600365 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa858010460 con 0x7fa85c0fed20 2023-07-14T17:44:05.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.917+0000 7fa8537fe700 1 --2- 172.21.15.88:0/685600365 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa8480a1220 0x7fa8480a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:05.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.917+0000 7fa861b9c700 1 --2- 172.21.15.88:0/685600365 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa8480a1220 0x7fa8480a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:05.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.917+0000 7fa8537fe700 1 -- 172.21.15.88:0/685600365 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa8580cd5d0 con 0x7fa85c0fed20 2023-07-14T17:44:05.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.917+0000 7fa861b9c700 1 --2- 172.21.15.88:0/685600365 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa8480a1220 0x7fa8480a36d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa85c0ffdf0 tx=0x7fa84c0096c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:05.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:05.920+0000 7fa8537fe700 1 -- 172.21.15.88:0/685600365 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa858096850 con 0x7fa85c0fed20 2023-07-14T17:44:06.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.113+0000 7fa864601700 1 -- 172.21.15.88:0/685600365 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa85c1a06e0 con 0x7fa85c0fed20 2023-07-14T17:44:06.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.115+0000 7fa8537fe700 1 -- 172.21.15.88:0/685600365 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24512 (secure 0 0 0) 0x7fa858010710 con 0x7fa85c0fed20 2023-07-14T17:44:06.097 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:06.098 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":88,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:43:44.887959+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:44:06.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.117+0000 7fa864601700 1 -- 172.21.15.88:0/685600365 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa8480a1220 msgr2=0x7fa8480a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.117+0000 7fa864601700 1 --2- 172.21.15.88:0/685600365 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa8480a1220 0x7fa8480a36d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa85c0ffdf0 tx=0x7fa84c0096c0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.117+0000 7fa864601700 1 -- 172.21.15.88:0/685600365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa85c0fed20 msgr2=0x7fa85c102ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.117+0000 7fa864601700 1 --2- 172.21.15.88:0/685600365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa85c0fed20 0x7fa85c102ad0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa85800d980 tx=0x7fa85800dd40 comp rx=0 tx=0).stop 2023-07-14T17:44:06.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.117+0000 7fa864601700 1 -- 172.21.15.88:0/685600365 shutdown_connections 2023-07-14T17:44:06.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.117+0000 7fa864601700 1 --2- 172.21.15.88:0/685600365 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fa85c1000d0 0x7fa85c101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.117+0000 7fa864601700 1 --2- 172.21.15.88:0/685600365 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fa8480a1220 0x7fa8480a36d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.117+0000 7fa864601700 1 --2- 172.21.15.88:0/685600365 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa85c0fed20 0x7fa85c102ad0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.117+0000 7fa864601700 1 --2- 172.21.15.88:0/685600365 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fa85c0ff6d0 0x7fa85c101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.117+0000 7fa864601700 1 -- 172.21.15.88:0/685600365 >> 172.21.15.88:0/685600365 conn(0x7fa85c0f98a0 msgr2=0x7fa85c0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:06.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.117+0000 7fa864601700 1 -- 172.21.15.88:0/685600365 shutdown_connections 2023-07-14T17:44:06.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.117+0000 7fa864601700 1 -- 172.21.15.88:0/685600365 wait complete. 2023-07-14T17:44:06.110 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:05 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/242180564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:44:06.110 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:05 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3700598604' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:44:06.111 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:05 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/820846377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:06.113 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:44:06.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:05 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/242180564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:44:06.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:05 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3700598604' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:44:06.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:05 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/820846377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:06.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.252+0000 7feee9170700 1 -- 172.21.15.88:0/2312247411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feee40fedc0 msgr2=0x7feee40ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.252+0000 7feee9170700 1 --2- 172.21.15.88:0/2312247411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feee40fedc0 0x7feee40ff210 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7feed4009990 tx=0x7feed4009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.252+0000 7feee9170700 1 -- 172.21.15.88:0/2312247411 shutdown_connections 2023-07-14T17:44:06.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.252+0000 7feee9170700 1 --2- 172.21.15.88:0/2312247411 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feee40ff750 0x7feee4107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.252+0000 7feee9170700 1 --2- 172.21.15.88:0/2312247411 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feee4100090 0x7feee41004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.252+0000 7feee9170700 1 --2- 172.21.15.88:0/2312247411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feee40fedc0 0x7feee40ff210 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.252+0000 7feee9170700 1 -- 172.21.15.88:0/2312247411 >> 172.21.15.88:0/2312247411 conn(0x7feee40f98a0 msgr2=0x7feee40fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:06.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.252+0000 7feee9170700 1 -- 172.21.15.88:0/2312247411 shutdown_connections 2023-07-14T17:44:06.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.252+0000 7feee9170700 1 -- 172.21.15.88:0/2312247411 wait complete. 2023-07-14T17:44:06.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.253+0000 7feee9170700 1 Processor -- start 2023-07-14T17:44:06.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.253+0000 7feee9170700 1 -- start start 2023-07-14T17:44:06.237 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.253+0000 7feee9170700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feee40fedc0 0x7feee4194770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.253+0000 7feee9170700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feee40ff750 0x7feee4194cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.253+0000 7feee9170700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feee4100090 0x7feee419bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.253+0000 7feee9170700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee419c2f0 con 0x7feee40fedc0 2023-07-14T17:44:06.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.253+0000 7feee9170700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee419c430 con 0x7feee4100090 2023-07-14T17:44:06.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.253+0000 7feee9170700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee419c5c0 con 0x7feee40ff750 2023-07-14T17:44:06.238 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.253+0000 7feee2d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feee40fedc0 0x7feee4194770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.253+0000 7feee2d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feee40fedc0 0x7feee4194770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42696/0 (socket says 172.21.15.88:42696) 2023-07-14T17:44:06.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.253+0000 7feee2d9d700 1 -- 172.21.15.88:0/2242484567 learned_addr learned my addr 172.21.15.88:0/2242484567 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:06.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.253+0000 7feee259c700 1 --2- 172.21.15.88:0/2242484567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feee40ff750 0x7feee4194cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.253+0000 7feee359e700 1 --2- 172.21.15.88:0/2242484567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feee4100090 0x7feee419bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.254+0000 7feee2d9d700 1 -- 172.21.15.88:0/2242484567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feee40ff750 msgr2=0x7feee4194cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.254+0000 7feee2d9d700 1 --2- 172.21.15.88:0/2242484567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feee40ff750 0x7feee4194cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.239 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.254+0000 7feee2d9d700 1 -- 172.21.15.88:0/2242484567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feee4100090 msgr2=0x7feee419bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.254+0000 7feee2d9d700 1 --2- 172.21.15.88:0/2242484567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feee4100090 0x7feee419bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.254+0000 7feee2d9d700 1 -- 172.21.15.88:0/2242484567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed40095f0 con 0x7feee40fedc0 2023-07-14T17:44:06.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.254+0000 7feee2d9d700 1 --2- 172.21.15.88:0/2242484567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feee40fedc0 0x7feee4194770 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7feed800d980 tx=0x7feed800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.254+0000 7feecbfff700 1 -- 172.21.15.88:0/2242484567 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed8004020 con 0x7feee40fedc0 2023-07-14T17:44:06.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.254+0000 7feecbfff700 1 -- 172.21.15.88:0/2242484567 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7feed800f690 con 0x7feee40fedc0 2023-07-14T17:44:06.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.254+0000 7feee9170700 1 -- 172.21.15.88:0/2242484567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee419c8a0 con 0x7feee40fedc0 2023-07-14T17:44:06.240 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.254+0000 7feecbfff700 1 -- 172.21.15.88:0/2242484567 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed80039e0 con 0x7feee40fedc0 2023-07-14T17:44:06.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.254+0000 7feee9170700 1 -- 172.21.15.88:0/2242484567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee419cdb0 con 0x7feee40fedc0 2023-07-14T17:44:06.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.255+0000 7feee9170700 1 -- 172.21.15.88:0/2242484567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee4106220 con 0x7feee40fedc0 2023-07-14T17:44:06.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.256+0000 7feecbfff700 1 -- 172.21.15.88:0/2242484567 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7feed800f800 con 0x7feee40fedc0 2023-07-14T17:44:06.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.256+0000 7feecbfff700 1 --2- 172.21.15.88:0/2242484567 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7feecc0a1220 0x7feecc0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.257+0000 7feee259c700 1 --2- 172.21.15.88:0/2242484567 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7feecc0a1220 0x7feecc0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.257+0000 7feecbfff700 1 -- 172.21.15.88:0/2242484567 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7feed80ccc60 con 0x7feee40fedc0 2023-07-14T17:44:06.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.257+0000 7feee259c700 1 --2- 172.21.15.88:0/2242484567 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7feecc0a1220 0x7feecc0a36d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7feed400b5c0 tx=0x7feed4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.242 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.260+0000 7feecbfff700 1 -- 172.21.15.88:0/2242484567 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7feed8095f40 con 0x7feee40fedc0 2023-07-14T17:44:06.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:05 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/242180564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:44:06.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:05 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3700598604' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:44:06.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:05 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/820846377' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:06.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.492+0000 7feee9170700 1 -- 172.21.15.88:0/2242484567 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feee402c8e0 con 0x7feee40fedc0 2023-07-14T17:44:06.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.493+0000 7feecbfff700 1 -- 172.21.15.88:0/2242484567 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10885 (secure 0 0 0) 0x7feed8099560 con 0x7feee40fedc0 2023-07-14T17:44:06.475 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:06.475 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T17:42:42.314442+0000","modified":"2023-07-14T17:43:04.774381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24473,"mds_2":24469,"mds_3":24454,"mds_4":24449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6839/3046592155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3046592155},{"type":"v1","addr":"172.21.15.88:6839","nonce":3046592155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6841/2509502918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6840","nonce":2509502918},{"type":"v1","addr":"172.21.15.88:6841","nonce":2509502918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6835/2332025068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":2332025068},{"type":"v1","addr":"172.21.15.178:6835","nonce":2332025068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24449":{"gid":24449,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.107:6835/3062637672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3062637672},{"type":"v1","addr":"172.21.15.107:6835","nonce":3062637672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.178:6837/3950404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3950404084},{"type":"v1","addr":"172.21.15.178:6837","nonce":3950404084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.178:6839/2953415077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2953415077},{"type":"v1","addr":"172.21.15.178:6839","nonce":2953415077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/1152608615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1152608615},{"type":"v1","addr":"172.21.15.107:6837","nonce":1152608615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6839/31646548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":31646548},{"type":"v1","addr":"172.21.15.107:6839","nonce":31646548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T17:44:06.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.495+0000 7feee9170700 1 -- 172.21.15.88:0/2242484567 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7feecc0a1220 msgr2=0x7feecc0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.495+0000 7feee9170700 1 --2- 172.21.15.88:0/2242484567 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7feecc0a1220 0x7feecc0a36d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7feed400b5c0 tx=0x7feed4005fb0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.495+0000 7feee9170700 1 -- 172.21.15.88:0/2242484567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feee40fedc0 msgr2=0x7feee4194770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.495+0000 7feee9170700 1 --2- 172.21.15.88:0/2242484567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feee40fedc0 0x7feee4194770 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7feed800d980 tx=0x7feed800dc90 comp rx=0 tx=0).stop 2023-07-14T17:44:06.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.495+0000 7feee9170700 1 -- 172.21.15.88:0/2242484567 shutdown_connections 2023-07-14T17:44:06.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.495+0000 7feee9170700 1 --2- 172.21.15.88:0/2242484567 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7feee4100090 0x7feee419bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.496+0000 7feee9170700 1 --2- 172.21.15.88:0/2242484567 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7feecc0a1220 0x7feecc0a36d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.496+0000 7feee9170700 1 --2- 172.21.15.88:0/2242484567 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7feee40fedc0 0x7feee4194770 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.496+0000 7feee9170700 1 --2- 172.21.15.88:0/2242484567 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7feee40ff750 0x7feee4194cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.496+0000 7feee9170700 1 -- 172.21.15.88:0/2242484567 >> 172.21.15.88:0/2242484567 conn(0x7feee40f98a0 msgr2=0x7feee4103360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:06.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.496+0000 7feee9170700 1 -- 172.21.15.88:0/2242484567 shutdown_connections 2023-07-14T17:44:06.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.496+0000 7feee9170700 1 -- 172.21.15.88:0/2242484567 wait complete. 2023-07-14T17:44:06.479 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 27 2023-07-14T17:44:06.490 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-07-14T17:42:42.314442+0000', 'modified': '2023-07-14T17:43:04.774381+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24506, 'mds_1': 24473, 'mds_2': 24469, 'mds_3': 24454, 'mds_4': 24449}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14616': {'gid': 14616, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.88:6839/3046592155', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6838', 'nonce': 3046592155}, {'type': 'v1', 'addr': '172.21.15.88:6839', 'nonce': 3046592155}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_14628': {'gid': 14628, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.88:6841/2509502918', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6840', 'nonce': 2509502918}, {'type': 'v1', 'addr': '172.21.15.88:6841', 'nonce': 2509502918}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24415': {'gid': 24415, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.178:6833/1716372869', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6832', 'nonce': 1716372869}, {'type': 'v1', 'addr': '172.21.15.178:6833', 'nonce': 1716372869}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24419': {'gid': 24419, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.107:6833/2946270337', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6832', 'nonce': 2946270337}, {'type': 'v1', 'addr': '172.21.15.107:6833', 'nonce': 2946270337}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.178:6835/2332025068', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6834', 'nonce': 2332025068}, {'type': 'v1', 'addr': '172.21.15.178:6835', 'nonce': 2332025068}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24449': {'gid': 24449, 'name': 'e', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.107:6835/3062637672', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6834', 'nonce': 3062637672}, {'type': 'v1', 'addr': '172.21.15.107:6835', 'nonce': 3062637672}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.178:6837/3950404084', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6836', 'nonce': 3950404084}, {'type': 'v1', 'addr': '172.21.15.178:6837', 'nonce': 3950404084}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24469': {'gid': 24469, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.178:6839/2953415077', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6838', 'nonce': 2953415077}, {'type': 'v1', 'addr': '172.21.15.178:6839', 'nonce': 2953415077}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24473': {'gid': 24473, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.107:6837/1152608615', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6836', 'nonce': 1152608615}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 1152608615}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24506': {'gid': 24506, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.107:6839/31646548', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6838', 'nonce': 31646548}, {'type': 'v1', 'addr': '172.21.15.107:6839', 'nonce': 31646548}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-07-14T17:44:06.490 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2023-07-14T17:44:06.490 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-07-14T17:44:06.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.633+0000 7f6c329a0700 1 -- 172.21.15.88:0/901956184 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c2c0fe280 msgr2=0x7f6c2c0fe690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.633+0000 7f6c329a0700 1 --2- 172.21.15.88:0/901956184 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c2c0fe280 0x7f6c2c0fe690 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6c20009990 tx=0x7f6c20009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.633+0000 7f6c329a0700 1 -- 172.21.15.88:0/901956184 shutdown_connections 2023-07-14T17:44:06.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.633+0000 7f6c329a0700 1 --2- 172.21.15.88:0/901956184 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c2c0ffe00 0x7f6c2c1066c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.633+0000 7f6c329a0700 1 --2- 172.21.15.88:0/901956184 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c2c0fe280 0x7f6c2c0fe690 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.633+0000 7f6c329a0700 1 --2- 172.21.15.88:0/901956184 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6c2c0ff470 0x7f6c2c0ff8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.633+0000 7f6c329a0700 1 -- 172.21.15.88:0/901956184 >> 172.21.15.88:0/901956184 conn(0x7f6c2c0f98a0 msgr2=0x7f6c2c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:06.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.633+0000 7f6c329a0700 1 -- 172.21.15.88:0/901956184 shutdown_connections 2023-07-14T17:44:06.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.634+0000 7f6c329a0700 1 -- 172.21.15.88:0/901956184 wait complete. 2023-07-14T17:44:06.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.634+0000 7f6c329a0700 1 Processor -- start 2023-07-14T17:44:06.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.634+0000 7f6c329a0700 1 -- start start 2023-07-14T17:44:06.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c329a0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c2c0fe280 0x7f6c2c1946b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c329a0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c2c0ff470 0x7f6c2c194bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c329a0700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6c2c0ffe00 0x7f6c2c19bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c329a0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c19c230 con 0x7f6c2c0ff470 2023-07-14T17:44:06.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c329a0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c19c370 con 0x7f6c2c0fe280 2023-07-14T17:44:06.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c329a0700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2c19c500 con 0x7f6c2c0ffe00 2023-07-14T17:44:06.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c2b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c2c0ff470 0x7f6c2c194bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c2b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c2c0ff470 0x7f6c2c194bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42712/0 (socket says 172.21.15.88:42712) 2023-07-14T17:44:06.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c2b7fe700 1 -- 172.21.15.88:0/1563122035 learned_addr learned my addr 172.21.15.88:0/1563122035 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:06.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c2bfff700 1 --2- 172.21.15.88:0/1563122035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c2c0fe280 0x7f6c2c1946b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c30f3d700 1 --2- 172.21.15.88:0/1563122035 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6c2c0ffe00 0x7f6c2c19bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c2b7fe700 1 -- 172.21.15.88:0/1563122035 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6c2c0ffe00 msgr2=0x7f6c2c19bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c2b7fe700 1 --2- 172.21.15.88:0/1563122035 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6c2c0ffe00 0x7f6c2c19bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c2b7fe700 1 -- 172.21.15.88:0/1563122035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c2c0fe280 msgr2=0x7f6c2c1946b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c2b7fe700 1 --2- 172.21.15.88:0/1563122035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c2c0fe280 0x7f6c2c1946b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c2b7fe700 1 -- 172.21.15.88:0/1563122035 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c200095f0 con 0x7f6c2c0ff470 2023-07-14T17:44:06.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c2b7fe700 1 --2- 172.21.15.88:0/1563122035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c2c0ff470 0x7f6c2c194bf0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c00c9b0 tx=0x7f6c1c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c297fa700 1 -- 172.21.15.88:0/1563122035 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c007b00 con 0x7f6c2c0ff470 2023-07-14T17:44:06.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c297fa700 1 -- 172.21.15.88:0/1563122035 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6c1c007c60 con 0x7f6c2c0ff470 2023-07-14T17:44:06.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c329a0700 1 -- 172.21.15.88:0/1563122035 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c2c19c7e0 con 0x7f6c2c0ff470 2023-07-14T17:44:06.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.635+0000 7f6c297fa700 1 -- 172.21.15.88:0/1563122035 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c018660 con 0x7f6c2c0ff470 2023-07-14T17:44:06.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.636+0000 7f6c329a0700 1 -- 172.21.15.88:0/1563122035 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c2c19ccf0 con 0x7f6c2c0ff470 2023-07-14T17:44:06.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.637+0000 7f6c297fa700 1 -- 172.21.15.88:0/1563122035 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6c1c00f450 con 0x7f6c2c0ff470 2023-07-14T17:44:06.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.638+0000 7f6c297fa700 1 --2- 172.21.15.88:0/1563122035 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6c140a11e0 0x7f6c140a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.638+0000 7f6c2bfff700 1 --2- 172.21.15.88:0/1563122035 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6c140a11e0 0x7f6c140a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.638+0000 7f6c297fa700 1 -- 172.21.15.88:0/1563122035 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6c1c0cce60 con 0x7f6c2c0ff470 2023-07-14T17:44:06.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.638+0000 7f6c2bfff700 1 --2- 172.21.15.88:0/1563122035 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6c140a11e0 0x7f6c140a3690 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000b5c0 tx=0x7f6c20005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.627 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.639+0000 7f6c3eb38b80 1 Processor -- start 2023-07-14T17:44:06.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.639+0000 7f6c3eb38b80 1 -- start start 2023-07-14T17:44:06.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.639+0000 7f6c3eb38b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5485f10 0x55d3a5486320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.639+0000 7f6c3eb38b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5486860 0x55d3a5486e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.639+0000 7f6c3eb38b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5487460 0x55d3a54878d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.639+0000 7f6c3eb38b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a538ac60 con 0x55d3a5485f10 2023-07-14T17:44:06.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.639+0000 7f6c3eb38b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a53dd9f0 con 0x55d3a5486860 2023-07-14T17:44:06.628 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.639+0000 7f6c3eb38b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a53d6e80 con 0x55d3a5487460 2023-07-14T17:44:06.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.639+0000 7f6c2bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5485f10 0x55d3a5486320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.639+0000 7f6c2bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5485f10 0x55d3a5486320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42718/0 (socket says 172.21.15.88:42718) 2023-07-14T17:44:06.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.639+0000 7f6c2bfff700 1 -- 172.21.15.88:0/2300705373 learned_addr learned my addr 172.21.15.88:0/2300705373 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:06.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.639+0000 7f6c2b7fe700 1 --2- 172.21.15.88:0/2300705373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5486860 0x55d3a5486e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.639+0000 7f6c2bfff700 1 -- 172.21.15.88:0/2300705373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5487460 msgr2=0x55d3a54878d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.639+0000 7f6c2bfff700 1 --2- 172.21.15.88:0/2300705373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5487460 0x55d3a54878d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.629 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.639+0000 7f6c2bfff700 1 -- 172.21.15.88:0/2300705373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5486860 msgr2=0x55d3a5486e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.639+0000 7f6c2bfff700 1 --2- 172.21.15.88:0/2300705373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5486860 0x55d3a5486e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.639+0000 7f6c2bfff700 1 -- 172.21.15.88:0/2300705373 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3a5204d90 con 0x55d3a5485f10 2023-07-14T17:44:06.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.639+0000 7f6c2b7fe700 1 --2- 172.21.15.88:0/2300705373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5486860 0x55d3a5486e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:06.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.640+0000 7f6c2bfff700 1 --2- 172.21.15.88:0/2300705373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5485f10 0x55d3a5486320 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c0ff2d0 tx=0x7f6c20020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=becf58a1424d1419 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.640+0000 7f6c127fc700 1 -- 172.21.15.88:0/2300705373 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2002d070 con 0x55d3a5485f10 2023-07-14T17:44:06.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.640+0000 7f6c127fc700 1 -- 172.21.15.88:0/2300705373 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6c2001e040 con 0x55d3a5485f10 2023-07-14T17:44:06.630 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.640+0000 7f6c127fc700 1 -- 172.21.15.88:0/2300705373 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c20004850 con 0x55d3a5485f10 2023-07-14T17:44:06.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.640+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/2300705373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5485f10 msgr2=0x55d3a5486320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.640+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/2300705373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5485f10 0x55d3a5486320 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c0ff2d0 tx=0x7f6c20020000 comp rx=0 tx=0).stop 2023-07-14T17:44:06.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.640+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/2300705373 shutdown_connections 2023-07-14T17:44:06.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.640+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/2300705373 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5487460 0x55d3a54878d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.640+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/2300705373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5485f10 0x55d3a5486320 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.640+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/2300705373 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5486860 0x55d3a5486e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.631 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.640+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/2300705373 >> 172.21.15.88:0/2300705373 conn(0x55d3a547b630 msgr2=0x55d3a5480980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:06.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.640+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/2300705373 shutdown_connections 2023-07-14T17:44:06.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.640+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/2300705373 wait complete. 2023-07-14T17:44:06.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.641+0000 7f6c3eb38b80 1 -- start start 2023-07-14T17:44:06.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.641+0000 7f6c3eb38b80 1 Processor -- start 2023-07-14T17:44:06.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.641+0000 7f6c3eb38b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5485f10 0x55d3a548f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.641+0000 7f6c3eb38b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5486860 0x55d3a548fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.632 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.641+0000 7f6c3eb38b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5487460 0x55d3a5496e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.641+0000 7f6c3eb38b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a5376a70 con 0x55d3a5487460 2023-07-14T17:44:06.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.641+0000 7f6c3eb38b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a5376770 con 0x55d3a5485f10 2023-07-14T17:44:06.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.641+0000 7f6c3eb38b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a525a8d0 con 0x55d3a5486860 2023-07-14T17:44:06.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.641+0000 7f6c3eb38b80 10 client.0 _ll_register_callbacks cb 0x55d3a5374710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:06.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.641+0000 7f6c30f3d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5487460 0x55d3a5496e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.641+0000 7f6c30f3d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5487460 0x55d3a5496e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42728/0 (socket says 172.21.15.88:42728) 2023-07-14T17:44:06.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.641+0000 7f6c30f3d700 1 -- 172.21.15.88:0/2282849403 learned_addr learned my addr 172.21.15.88:0/2282849403 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:06.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.641+0000 7f6c2bfff700 1 --2- 172.21.15.88:0/2282849403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5485f10 0x55d3a548f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c2b7fe700 1 --2- 172.21.15.88:0/2282849403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5486860 0x55d3a548fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c30f3d700 1 -- 172.21.15.88:0/2282849403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5486860 msgr2=0x55d3a548fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c30f3d700 1 --2- 172.21.15.88:0/2282849403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5486860 0x55d3a548fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c30f3d700 1 -- 172.21.15.88:0/2282849403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5485f10 msgr2=0x55d3a548f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c30f3d700 1 --2- 172.21.15.88:0/2282849403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5485f10 0x55d3a548f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c30f3d700 1 -- 172.21.15.88:0/2282849403 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c2c066f10 con 0x55d3a5487460 2023-07-14T17:44:06.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c2b7fe700 1 --2- 172.21.15.88:0/2282849403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5486860 0x55d3a548fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:06.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c2bfff700 1 --2- 172.21.15.88:0/2282849403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5485f10 0x55d3a548f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T17:44:06.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c30f3d700 1 --2- 172.21.15.88:0/2282849403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5487460 0x55d3a5496e10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400e930 tx=0x7f6c2400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35153b34356fb46d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c0b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:06.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c0b7fe700 1 -- 172.21.15.88:0/2282849403 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c24019070 con 0x55d3a5487460 2023-07-14T17:44:06.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c0b7fe700 1 -- 172.21.15.88:0/2282849403 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6c2400ee80 con 0x55d3a5487460 2023-07-14T17:44:06.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/2282849403 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d3a53304a0 con 0x55d3a5487460 2023-07-14T17:44:06.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c0b7fe700 1 -- 172.21.15.88:0/2282849403 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c24007af0 con 0x55d3a5487460 2023-07-14T17:44:06.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c3eb38b80 10 client.15231 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:06.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/2282849403 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d3a51edc10 con 0x55d3a5487460 2023-07-14T17:44:06.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c0b7fe700 1 -- 172.21.15.88:0/2282849403 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55d3a53304a0 con 0x55d3a5487460 2023-07-14T17:44:06.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c0b7fe700 1 -- 172.21.15.88:0/2282849403 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6c2401c020 con 0x55d3a5487460 2023-07-14T17:44:06.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c3eb38b80 10 client.15231 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:06.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.642+0000 7f6c3eb38b80 10 client.15231 resolve_mds: resolved name 'e' to daemon mds.e 2023-07-14T17:44:06.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.643+0000 7f6c3eb38b80 20 client.15231 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:06.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.643+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/2282849403 >> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] conn(0x55d3a549ac10 0x55d3a5499fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.643+0000 7f6c3eb38b80 4 client.15231 mds_command: new command op to 24449 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T17:44:06.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.643+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/2282849403 --> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d3a5454d50 con 0x55d3a549ac10 2023-07-14T17:44:06.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.643+0000 7f6c2bfff700 1 --2- 172.21.15.88:0/2282849403 >> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] conn(0x55d3a549ac10 0x55d3a5499fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.643+0000 7f6c0b7fe700 10 client.15231 ms_handle_connect on v2:172.21.15.107:6834/3062637672 2023-07-14T17:44:06.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.643+0000 7f6c2bfff700 1 --2- 172.21.15.88:0/2282849403 >> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] conn(0x55d3a549ac10 0x55d3a5499fc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fb22c384621d258a server_cookie=7aaa0a547871fffe in_seq=0 out_seq=0 2023-07-14T17:44:06.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.644+0000 7f6c0b7fe700 1 -- 172.21.15.88:0/2282849403 <== mds.4 v2:172.21.15.107:6834/3062637672 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55d3a5454d50 con 0x55d3a549ac10 2023-07-14T17:44:06.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.644+0000 7f6c0b7fe700 10 client.15231 handle_command_reply: tid=0 2023-07-14T17:44:06.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.644+0000 7f6c3eb38b80 1 client.15231 shutdown 2023-07-14T17:44:06.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.644+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/2282849403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5487460 msgr2=0x55d3a5496e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.644+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/2282849403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5487460 0x55d3a5496e10 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6c2400e930 tx=0x7f6c2400ec40 comp rx=0 tx=0).stop 2023-07-14T17:44:06.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.644+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/2282849403 shutdown_connections 2023-07-14T17:44:06.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.644+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/2282849403 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5487460 0x55d3a5496e10 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.644+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/2282849403 >> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] conn(0x55d3a549ac10 0x55d3a5499fc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.644+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/2282849403 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5485f10 0x55d3a548f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.644+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/2282849403 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5486860 0x55d3a548fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.644+0000 7f6c0b7fe700 0 client.15231 ms_handle_reset on v2:172.21.15.107:6834/3062637672 2023-07-14T17:44:06.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.644+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/2282849403 >> 172.21.15.88:0/2282849403 conn(0x55d3a547b630 msgr2=0x55d3a547f600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:06.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.644+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/2282849403 shutdown_connections 2023-07-14T17:44:06.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.644+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/2282849403 wait complete. 2023-07-14T17:44:06.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.645+0000 7f6c3eb38b80 20 client.15231 trim_cache size 0 max 16384 2023-07-14T17:44:06.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.662+0000 7f6c3eb38b80 1 Processor -- start 2023-07-14T17:44:06.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.662+0000 7f6c3eb38b80 1 -- start start 2023-07-14T17:44:06.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.662+0000 7f6c3eb38b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5485f10 0x55d3a54800a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.662+0000 7f6c3eb38b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5486860 0x55d3a548d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.662+0000 7f6c3eb38b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5487460 0x55d3a5499fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.662+0000 7f6c3eb38b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a548f460 con 0x55d3a5485f10 2023-07-14T17:44:06.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.662+0000 7f6c3eb38b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a53304a0 con 0x55d3a5487460 2023-07-14T17:44:06.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.662+0000 7f6c3eb38b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a5490a80 con 0x55d3a5486860 2023-07-14T17:44:06.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.662+0000 7f6c2bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5485f10 0x55d3a54800a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.662+0000 7f6c2bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5485f10 0x55d3a54800a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42732/0 (socket says 172.21.15.88:42732) 2023-07-14T17:44:06.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.662+0000 7f6c2bfff700 1 -- 172.21.15.88:0/2437306139 learned_addr learned my addr 172.21.15.88:0/2437306139 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:06.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.662+0000 7f6c2b7fe700 1 --2- 172.21.15.88:0/2437306139 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5486860 0x55d3a548d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.662+0000 7f6c30f3d700 1 --2- 172.21.15.88:0/2437306139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5487460 0x55d3a5499fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.662+0000 7f6c2bfff700 1 -- 172.21.15.88:0/2437306139 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5486860 msgr2=0x55d3a548d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.662+0000 7f6c2bfff700 1 --2- 172.21.15.88:0/2437306139 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5486860 0x55d3a548d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.662+0000 7f6c2bfff700 1 -- 172.21.15.88:0/2437306139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5487460 msgr2=0x55d3a5499fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.662+0000 7f6c2bfff700 1 --2- 172.21.15.88:0/2437306139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5487460 0x55d3a5499fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.663+0000 7f6c2bfff700 1 -- 172.21.15.88:0/2437306139 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3a522b200 con 0x55d3a5485f10 2023-07-14T17:44:06.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.663+0000 7f6c2bfff700 1 --2- 172.21.15.88:0/2437306139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5485f10 0x55d3a54800a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6c20006010 tx=0x7f6c20023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164c981a23f99e0a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.663+0000 7f6c12ffd700 1 -- 172.21.15.88:0/2437306139 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2002d040 con 0x55d3a5485f10 2023-07-14T17:44:06.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.663+0000 7f6c12ffd700 1 -- 172.21.15.88:0/2437306139 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6c2000bce0 con 0x55d3a5485f10 2023-07-14T17:44:06.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.663+0000 7f6c12ffd700 1 -- 172.21.15.88:0/2437306139 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c20004c00 con 0x55d3a5485f10 2023-07-14T17:44:06.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.663+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/2437306139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5485f10 msgr2=0x55d3a54800a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.663+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/2437306139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5485f10 0x55d3a54800a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6c20006010 tx=0x7f6c20023000 comp rx=0 tx=0).stop 2023-07-14T17:44:06.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.663+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/2437306139 shutdown_connections 2023-07-14T17:44:06.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.663+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/2437306139 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5487460 0x55d3a5499fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.663+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/2437306139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5485f10 0x55d3a54800a0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.663+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/2437306139 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5486860 0x55d3a548d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.663+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/2437306139 >> 172.21.15.88:0/2437306139 conn(0x55d3a547b630 msgr2=0x55d3a548e560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:06.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.663+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/2437306139 shutdown_connections 2023-07-14T17:44:06.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.663+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/2437306139 wait complete. 2023-07-14T17:44:06.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.664+0000 7f6c3eb38b80 1 -- start start 2023-07-14T17:44:06.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.664+0000 7f6c3eb38b80 1 Processor -- start 2023-07-14T17:44:06.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.664+0000 7f6c3eb38b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5485f10 0x55d3a54ed8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.664+0000 7f6c3eb38b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5486860 0x55d3a54efe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.664+0000 7f6c3eb38b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5487460 0x55d3a54f2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.664+0000 7f6c3eb38b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a54f2890 con 0x55d3a5486860 2023-07-14T17:44:06.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.664+0000 7f6c3eb38b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a54f29d0 con 0x55d3a5485f10 2023-07-14T17:44:06.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.664+0000 7f6c3eb38b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3a54f2b10 con 0x55d3a5487460 2023-07-14T17:44:06.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.664+0000 7f6c3eb38b80 10 client.0 _ll_register_callbacks cb 0x7f6c2400e1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:06.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.664+0000 7f6c2b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5486860 0x55d3a54efe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.664+0000 7f6c2b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5486860 0x55d3a54efe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42740/0 (socket says 172.21.15.88:42740) 2023-07-14T17:44:06.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.664+0000 7f6c2b7fe700 1 -- 172.21.15.88:0/1865203218 learned_addr learned my addr 172.21.15.88:0/1865203218 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:06.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.664+0000 7f6c30f3d700 1 --2- 172.21.15.88:0/1865203218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5487460 0x55d3a54f2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.664+0000 7f6c2bfff700 1 --2- 172.21.15.88:0/1865203218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5485f10 0x55d3a54ed8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c2b7fe700 1 -- 172.21.15.88:0/1865203218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5487460 msgr2=0x55d3a54f2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c2b7fe700 1 --2- 172.21.15.88:0/1865203218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5487460 0x55d3a54f2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c2b7fe700 1 -- 172.21.15.88:0/1865203218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5485f10 msgr2=0x55d3a54ed8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c2b7fe700 1 --2- 172.21.15.88:0/1865203218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5485f10 0x55d3a54ed8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c2b7fe700 1 -- 172.21.15.88:0/1865203218 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d3a5204d90 con 0x55d3a5486860 2023-07-14T17:44:06.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c2b7fe700 1 --2- 172.21.15.88:0/1865203218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5486860 0x55d3a54efe00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c099d30 tx=0x7f6c1c09ddd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f9cfe7e599da454 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c0b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:06.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c0b7fe700 1 -- 172.21.15.88:0/1865203218 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c098200 con 0x55d3a5486860 2023-07-14T17:44:06.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c0b7fe700 1 -- 172.21.15.88:0/1865203218 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6c1c0b1070 con 0x55d3a5486860 2023-07-14T17:44:06.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/1865203218 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6c2001e040 con 0x55d3a5486860 2023-07-14T17:44:06.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c0b7fe700 1 -- 172.21.15.88:0/1865203218 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c1c0ca170 con 0x55d3a5486860 2023-07-14T17:44:06.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c3eb38b80 10 client.15237 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:06.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/1865203218 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d3a5203260 con 0x55d3a5486860 2023-07-14T17:44:06.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c0b7fe700 1 -- 172.21.15.88:0/1865203218 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6c1c07c080 con 0x55d3a5486860 2023-07-14T17:44:06.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c0b7fe700 1 -- 172.21.15.88:0/1865203218 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6c1c0a1d60 con 0x55d3a5486860 2023-07-14T17:44:06.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c3eb38b80 10 client.15237 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:06.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c3eb38b80 10 client.15237 resolve_mds: resolved name 'e' to daemon mds.e 2023-07-14T17:44:06.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c3eb38b80 20 client.15237 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:06.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/1865203218 >> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] conn(0x55d3a54f78e0 0x55d3a54f4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c3eb38b80 4 client.15237 mds_command: new command op to 24449 tid=0[{"prefix": "status"}] 2023-07-14T17:44:06.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.665+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/1865203218 --> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d3a53cfe60 con 0x55d3a54f78e0 2023-07-14T17:44:06.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.666+0000 7f6c2bfff700 1 --2- 172.21.15.88:0/1865203218 >> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] conn(0x55d3a54f78e0 0x55d3a54f4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.666+0000 7f6c2bfff700 1 --2- 172.21.15.88:0/1865203218 >> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] conn(0x55d3a54f78e0 0x55d3a54f4270 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d8e9ae21c7ab2ad1 server_cookie=99f5c43b4477952b in_seq=0 out_seq=0 2023-07-14T17:44:06.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.666+0000 7f6c0b7fe700 10 client.15237 ms_handle_connect on v2:172.21.15.107:6834/3062637672 2023-07-14T17:44:06.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.666+0000 7f6c0b7fe700 1 -- 172.21.15.88:0/1865203218 <== mds.4 v2:172.21.15.107:6834/3062637672 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55d3a53cfe60 con 0x55d3a54f78e0 2023-07-14T17:44:06.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.666+0000 7f6c0b7fe700 10 client.15237 handle_command_reply: tid=0 2023-07-14T17:44:06.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.666+0000 7f6c3eb38b80 1 client.15237 shutdown 2023-07-14T17:44:06.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.666+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/1865203218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5486860 msgr2=0x55d3a54efe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.666+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/1865203218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5486860 0x55d3a54efe00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c099d30 tx=0x7f6c1c09ddd0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.666+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/1865203218 shutdown_connections 2023-07-14T17:44:06.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.666+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/1865203218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55d3a5487460 0x55d3a54f2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.666+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/1865203218 >> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] conn(0x55d3a54f78e0 0x55d3a54f4270 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.666+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/1865203218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55d3a5485f10 0x55d3a54ed8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.666+0000 7f6c0b7fe700 0 client.15237 ms_handle_reset on v2:172.21.15.107:6834/3062637672 2023-07-14T17:44:06.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.666+0000 7f6c3eb38b80 1 --2- 172.21.15.88:0/1865203218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55d3a5486860 0x55d3a54efe00 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.666+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/1865203218 >> 172.21.15.88:0/1865203218 conn(0x55d3a547b630 msgr2=0x55d3a548d880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:06.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.667+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/1865203218 shutdown_connections 2023-07-14T17:44:06.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.667+0000 7f6c3eb38b80 1 -- 172.21.15.88:0/1865203218 wait complete. 2023-07-14T17:44:06.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.667+0000 7f6c3eb38b80 20 client.15237 trim_cache size 0 max 16384 2023-07-14T17:44:06.659 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-07-14T17:44:06.659 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "b462a556-226c-11ee-9b33-001a4aab830c", 2023-07-14T17:44:06.660 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 4, 2023-07-14T17:44:06.660 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 24449, 2023-07-14T17:44:06.660 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2023-07-14T17:44:06.660 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2023-07-14T17:44:06.660 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2023-07-14T17:44:06.660 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 62.905514680000003, 2023-07-14T17:44:06.660 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 27, 2023-07-14T17:44:06.661 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 88, 2023-07-14T17:44:06.661 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 86, 2023-07-14T17:44:06.661 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 104.901964373 2023-07-14T17:44:06.661 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-07-14T17:44:06.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.667+0000 7f6c329a0700 1 -- 172.21.15.88:0/1563122035 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6c140a11e0 msgr2=0x7f6c140a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.667+0000 7f6c329a0700 1 --2- 172.21.15.88:0/1563122035 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6c140a11e0 0x7f6c140a3690 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000b5c0 tx=0x7f6c20005960 comp rx=0 tx=0).stop 2023-07-14T17:44:06.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.667+0000 7f6c329a0700 1 -- 172.21.15.88:0/1563122035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c2c0ff470 msgr2=0x7f6c2c194bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.667+0000 7f6c329a0700 1 --2- 172.21.15.88:0/1563122035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c2c0ff470 0x7f6c2c194bf0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c00c9b0 tx=0x7f6c1c00ccc0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.667+0000 7f6c329a0700 1 -- 172.21.15.88:0/1563122035 shutdown_connections 2023-07-14T17:44:06.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.667+0000 7f6c329a0700 1 --2- 172.21.15.88:0/1563122035 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6c2c0ffe00 0x7f6c2c19bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.667+0000 7f6c329a0700 1 --2- 172.21.15.88:0/1563122035 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6c2c0fe280 0x7f6c2c1946b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.667+0000 7f6c329a0700 1 --2- 172.21.15.88:0/1563122035 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6c140a11e0 0x7f6c140a3690 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.667+0000 7f6c329a0700 1 --2- 172.21.15.88:0/1563122035 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c2c0ff470 0x7f6c2c194bf0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.667+0000 7f6c329a0700 1 -- 172.21.15.88:0/1563122035 >> 172.21.15.88:0/1563122035 conn(0x7f6c2c0f98a0 msgr2=0x7f6c2c0fbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:06.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.667+0000 7f6c329a0700 1 -- 172.21.15.88:0/1563122035 shutdown_connections 2023-07-14T17:44:06.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.667+0000 7f6c329a0700 1 -- 172.21.15.88:0/1563122035 wait complete. 2023-07-14T17:44:06.664 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-07-14T17:44:06.787 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.804+0000 7f98bb5e0700 1 -- 172.21.15.88:0/3653995086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98b4100770 msgr2=0x7f98b4107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.804+0000 7f98bb5e0700 1 --2- 172.21.15.88:0/3653995086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98b4100770 0x7f98b4107010 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00b0a0 tx=0x7f98ac00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.804+0000 7f98bb5e0700 1 -- 172.21.15.88:0/3653995086 shutdown_connections 2023-07-14T17:44:06.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.804+0000 7f98bb5e0700 1 --2- 172.21.15.88:0/3653995086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98b4100770 0x7f98b4107010 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.804+0000 7f98bb5e0700 1 --2- 172.21.15.88:0/3653995086 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98b40feb30 0x7f98b40fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.804+0000 7f98bb5e0700 1 --2- 172.21.15.88:0/3653995086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b40ffdc0 0x7f98b4100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.804+0000 7f98bb5e0700 1 -- 172.21.15.88:0/3653995086 >> 172.21.15.88:0/3653995086 conn(0x7f98b40fa150 msgr2=0x7f98b40fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:06.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.805+0000 7f98bb5e0700 1 -- 172.21.15.88:0/3653995086 shutdown_connections 2023-07-14T17:44:06.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.805+0000 7f98bb5e0700 1 -- 172.21.15.88:0/3653995086 wait complete. 2023-07-14T17:44:06.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.805+0000 7f98bb5e0700 1 Processor -- start 2023-07-14T17:44:06.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.805+0000 7f98bb5e0700 1 -- start start 2023-07-14T17:44:06.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.805+0000 7f98bb5e0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b40feb30 0x7f98b40694e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.805+0000 7f98bb5e0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98b40ffdc0 0x7f98b419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.805+0000 7f98bb5e0700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98b4100770 0x7f98b419ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.805+0000 7f98bb5e0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b419d4c0 con 0x7f98b40ffdc0 2023-07-14T17:44:06.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.805+0000 7f98bb5e0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b419d630 con 0x7f98b40feb30 2023-07-14T17:44:06.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.805+0000 7f98bb5e0700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b419d7f0 con 0x7f98b4100770 2023-07-14T17:44:06.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.805+0000 7f98b8b7b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98b40ffdc0 0x7f98b419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.805+0000 7f98b8b7b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98b40ffdc0 0x7f98b419a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42754/0 (socket says 172.21.15.88:42754) 2023-07-14T17:44:06.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.805+0000 7f98b8b7b700 1 -- 172.21.15.88:0/4271950960 learned_addr learned my addr 172.21.15.88:0/4271950960 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:06.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.805+0000 7f98b937c700 1 --2- 172.21.15.88:0/4271950960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b40feb30 0x7f98b40694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.806+0000 7f98b9b7d700 1 --2- 172.21.15.88:0/4271950960 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98b4100770 0x7f98b419ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.806+0000 7f98b8b7b700 1 -- 172.21.15.88:0/4271950960 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98b4100770 msgr2=0x7f98b419ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.806+0000 7f98b8b7b700 1 --2- 172.21.15.88:0/4271950960 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98b4100770 0x7f98b419ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.806+0000 7f98b8b7b700 1 -- 172.21.15.88:0/4271950960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b40feb30 msgr2=0x7f98b40694e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.791 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.806+0000 7f98b8b7b700 1 --2- 172.21.15.88:0/4271950960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b40feb30 0x7f98b40694e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.806+0000 7f98b8b7b700 1 -- 172.21.15.88:0/4271950960 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac009d00 con 0x7f98b40ffdc0 2023-07-14T17:44:06.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.806+0000 7f98b9b7d700 1 --2- 172.21.15.88:0/4271950960 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98b4100770 0x7f98b419ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:06.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.806+0000 7f98b8b7b700 1 --2- 172.21.15.88:0/4271950960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98b40ffdc0 0x7f98b419a6a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f98b000d980 tx=0x7f98b000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.806+0000 7f98a67fc700 1 -- 172.21.15.88:0/4271950960 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b0004220 con 0x7f98b40ffdc0 2023-07-14T17:44:06.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.806+0000 7f98a67fc700 1 -- 172.21.15.88:0/4271950960 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f98b0007640 con 0x7f98b40ffdc0 2023-07-14T17:44:06.792 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.806+0000 7f98bb5e0700 1 -- 172.21.15.88:0/4271950960 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b419dad0 con 0x7f98b40ffdc0 2023-07-14T17:44:06.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.806+0000 7f98a67fc700 1 -- 172.21.15.88:0/4271950960 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98b00055e0 con 0x7f98b40ffdc0 2023-07-14T17:44:06.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.806+0000 7f98bb5e0700 1 -- 172.21.15.88:0/4271950960 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b419dfe0 con 0x7f98b40ffdc0 2023-07-14T17:44:06.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.808+0000 7f98a67fc700 1 -- 172.21.15.88:0/4271950960 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f98b0004750 con 0x7f98b40ffdc0 2023-07-14T17:44:06.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.809+0000 7f98a67fc700 1 --2- 172.21.15.88:0/4271950960 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f98a00a11e0 0x7f98a00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.809+0000 7f98b937c700 1 --2- 172.21.15.88:0/4271950960 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f98a00a11e0 0x7f98a00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.809+0000 7f98a67fc700 1 -- 172.21.15.88:0/4271950960 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f98b00ccbb0 con 0x7f98b40ffdc0 2023-07-14T17:44:06.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.810+0000 7f98b937c700 1 --2- 172.21.15.88:0/4271950960 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f98a00a11e0 0x7f98a00a3690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f98a8009c60 tx=0x7f98a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.810+0000 7f98c7778b80 1 Processor -- start 2023-07-14T17:44:06.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.810+0000 7f98c7778b80 1 -- start start 2023-07-14T17:44:06.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.810+0000 7f98c7778b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 0x562f3f801320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.810+0000 7f98c7778b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 0x562f3f801e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.810+0000 7f98c7778b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f802460 0x562f3f8028d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.810+0000 7f98c7778b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f3f705c60 con 0x562f3f800f10 2023-07-14T17:44:06.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.810+0000 7f98c7778b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f3f7589f0 con 0x562f3f801860 2023-07-14T17:44:06.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.810+0000 7f98c7778b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f3f751e80 con 0x562f3f802460 2023-07-14T17:44:06.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.811+0000 7f98b937c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 0x562f3f801320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.811+0000 7f98b937c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 0x562f3f801320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42760/0 (socket says 172.21.15.88:42760) 2023-07-14T17:44:06.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.811+0000 7f98b937c700 1 -- 172.21.15.88:0/1535209713 learned_addr learned my addr 172.21.15.88:0/1535209713 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:06.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.811+0000 7f98b8b7b700 1 --2- 172.21.15.88:0/1535209713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 0x562f3f801e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.811+0000 7f98b9b7d700 1 --2- 172.21.15.88:0/1535209713 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f802460 0x562f3f8028d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.811+0000 7f98b937c700 1 -- 172.21.15.88:0/1535209713 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f802460 msgr2=0x562f3f8028d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.811+0000 7f98b937c700 1 --2- 172.21.15.88:0/1535209713 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f802460 0x562f3f8028d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.811+0000 7f98b937c700 1 -- 172.21.15.88:0/1535209713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 msgr2=0x562f3f801e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.811+0000 7f98b937c700 1 --2- 172.21.15.88:0/1535209713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 0x562f3f801e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.811+0000 7f98b937c700 1 -- 172.21.15.88:0/1535209713 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f3f57fd90 con 0x562f3f800f10 2023-07-14T17:44:06.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.811+0000 7f98b8b7b700 1 --2- 172.21.15.88:0/1535209713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 0x562f3f801e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T17:44:06.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.811+0000 7f98b937c700 1 --2- 172.21.15.88:0/1535209713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 0x562f3f801320 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f98a800e4e0 tx=0x7f98a8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=538cbb072a640152 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.811+0000 7f989f7fe700 1 -- 172.21.15.88:0/1535209713 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a800ee00 con 0x562f3f800f10 2023-07-14T17:44:06.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.811+0000 7f989f7fe700 1 -- 172.21.15.88:0/1535209713 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f98a8021070 con 0x562f3f800f10 2023-07-14T17:44:06.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.811+0000 7f989f7fe700 1 -- 172.21.15.88:0/1535209713 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a801d720 con 0x562f3f800f10 2023-07-14T17:44:06.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.812+0000 7f98c7778b80 1 -- 172.21.15.88:0/1535209713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 msgr2=0x562f3f801320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.812+0000 7f98c7778b80 1 --2- 172.21.15.88:0/1535209713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 0x562f3f801320 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f98a800e4e0 tx=0x7f98a8014000 comp rx=0 tx=0).stop 2023-07-14T17:44:06.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.812+0000 7f98c7778b80 1 -- 172.21.15.88:0/1535209713 shutdown_connections 2023-07-14T17:44:06.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.812+0000 7f98c7778b80 1 --2- 172.21.15.88:0/1535209713 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f802460 0x562f3f8028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.812+0000 7f98c7778b80 1 --2- 172.21.15.88:0/1535209713 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 0x562f3f801320 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.812+0000 7f98c7778b80 1 --2- 172.21.15.88:0/1535209713 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 0x562f3f801e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.812+0000 7f98c7778b80 1 -- 172.21.15.88:0/1535209713 >> 172.21.15.88:0/1535209713 conn(0x562f3f7f6630 msgr2=0x562f3f7fb980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:06.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.812+0000 7f98c7778b80 1 -- 172.21.15.88:0/1535209713 shutdown_connections 2023-07-14T17:44:06.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.812+0000 7f98c7778b80 1 -- 172.21.15.88:0/1535209713 wait complete. 2023-07-14T17:44:06.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.812+0000 7f98c7778b80 1 -- start start 2023-07-14T17:44:06.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.812+0000 7f98c7778b80 1 Processor -- start 2023-07-14T17:44:06.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.813+0000 7f98c7778b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 0x562f3f80a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.813+0000 7f98c7778b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 0x562f3f80ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.813+0000 7f98c7778b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f802460 0x562f3f811e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.813+0000 7f98c7778b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f3f6f1a70 con 0x562f3f800f10 2023-07-14T17:44:06.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.813+0000 7f98c7778b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f3f6f1770 con 0x562f3f801860 2023-07-14T17:44:06.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.813+0000 7f98c7778b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f3f5d58d0 con 0x562f3f802460 2023-07-14T17:44:06.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.813+0000 7f98b937c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 0x562f3f80a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.813+0000 7f98c7778b80 10 client.0 _ll_register_callbacks cb 0x562f3f6ef710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:06.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.813+0000 7f98b937c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 0x562f3f80a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42766/0 (socket says 172.21.15.88:42766) 2023-07-14T17:44:06.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.813+0000 7f98b937c700 1 -- 172.21.15.88:0/156131026 learned_addr learned my addr 172.21.15.88:0/156131026 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:06.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.813+0000 7f98b8b7b700 1 --2- 172.21.15.88:0/156131026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 0x562f3f80ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.813+0000 7f98b9b7d700 1 --2- 172.21.15.88:0/156131026 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f802460 0x562f3f811e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.813+0000 7f98b937c700 1 -- 172.21.15.88:0/156131026 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f802460 msgr2=0x562f3f811e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.813+0000 7f98b937c700 1 --2- 172.21.15.88:0/156131026 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f802460 0x562f3f811e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.813+0000 7f98b937c700 1 -- 172.21.15.88:0/156131026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 msgr2=0x562f3f80ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.813+0000 7f98b937c700 1 --2- 172.21.15.88:0/156131026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 0x562f3f80ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.813+0000 7f98b937c700 1 -- 172.21.15.88:0/156131026 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b419dfe0 con 0x562f3f800f10 2023-07-14T17:44:06.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.814+0000 7f98b937c700 1 --2- 172.21.15.88:0/156131026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 0x562f3f80a8c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f98a800ee20 tx=0x7f98a800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f51de21d256f3ca server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.814+0000 7f989cff9700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:06.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.814+0000 7f989cff9700 1 -- 172.21.15.88:0/156131026 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a8005490 con 0x562f3f800f10 2023-07-14T17:44:06.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.814+0000 7f989cff9700 1 -- 172.21.15.88:0/156131026 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f98a80055f0 con 0x562f3f800f10 2023-07-14T17:44:06.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.814+0000 7f98c7778b80 1 -- 172.21.15.88:0/156131026 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562f3f6ab4a0 con 0x562f3f800f10 2023-07-14T17:44:06.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.814+0000 7f989cff9700 1 -- 172.21.15.88:0/156131026 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98a8003680 con 0x562f3f800f10 2023-07-14T17:44:06.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.814+0000 7f98c7778b80 10 client.15249 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:06.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.814+0000 7f98c7778b80 1 -- 172.21.15.88:0/156131026 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7f98a8017070 con 0x562f3f800f10 2023-07-14T17:44:06.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.814+0000 7f989cff9700 1 -- 172.21.15.88:0/156131026 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x562f3f6ab4a0 con 0x562f3f800f10 2023-07-14T17:44:06.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.814+0000 7f989cff9700 1 -- 172.21.15.88:0/156131026 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f98a801c890 con 0x562f3f800f10 2023-07-14T17:44:06.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.814+0000 7f98c7778b80 10 client.15249 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:06.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.814+0000 7f98c7778b80 10 client.15249 resolve_mds: resolved name 'c' to daemon mds.c 2023-07-14T17:44:06.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.814+0000 7f98c7778b80 20 client.15249 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:06.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.814+0000 7f98c7778b80 1 --2- 172.21.15.88:0/156131026 >> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] conn(0x562f3f815a80 0x562f3f814ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.814+0000 7f98c7778b80 4 client.15249 mds_command: new command op to 24454 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T17:44:06.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.814+0000 7f98c7778b80 1 -- 172.21.15.88:0/156131026 --> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562f3f7cfd50 con 0x562f3f815a80 2023-07-14T17:44:06.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.815+0000 7f98b8b7b700 1 --2- 172.21.15.88:0/156131026 >> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] conn(0x562f3f815a80 0x562f3f814ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.815+0000 7f989cff9700 10 client.15249 ms_handle_connect on v2:172.21.15.178:6836/3950404084 2023-07-14T17:44:06.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.815+0000 7f98b8b7b700 1 --2- 172.21.15.88:0/156131026 >> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] conn(0x562f3f815a80 0x562f3f814ee0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a855c6448d431729 server_cookie=bcba2b26d1e83d0d in_seq=0 out_seq=0 2023-07-14T17:44:06.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.816+0000 7f989cff9700 1 -- 172.21.15.88:0/156131026 <== mds.3 v2:172.21.15.178:6836/3950404084 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x562f3f7cfd50 con 0x562f3f815a80 2023-07-14T17:44:06.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.816+0000 7f989cff9700 10 client.15249 handle_command_reply: tid=0 2023-07-14T17:44:06.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.816+0000 7f98c7778b80 1 client.15249 shutdown 2023-07-14T17:44:06.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.816+0000 7f98c7778b80 1 -- 172.21.15.88:0/156131026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 msgr2=0x562f3f80a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.816+0000 7f98c7778b80 1 --2- 172.21.15.88:0/156131026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 0x562f3f80a8c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f98a800ee20 tx=0x7f98a800ee50 comp rx=0 tx=0).stop 2023-07-14T17:44:06.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.816+0000 7f98c7778b80 1 -- 172.21.15.88:0/156131026 shutdown_connections 2023-07-14T17:44:06.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.816+0000 7f98c7778b80 1 --2- 172.21.15.88:0/156131026 >> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] conn(0x562f3f815a80 0x562f3f814ee0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.816+0000 7f98c7778b80 1 --2- 172.21.15.88:0/156131026 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f802460 0x562f3f811e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.816+0000 7f98c7778b80 1 --2- 172.21.15.88:0/156131026 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 0x562f3f80a8c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.816+0000 7f98c7778b80 1 --2- 172.21.15.88:0/156131026 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 0x562f3f80ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.816+0000 7f989cff9700 0 client.15249 ms_handle_reset on v2:172.21.15.178:6836/3950404084 2023-07-14T17:44:06.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.816+0000 7f98c7778b80 1 -- 172.21.15.88:0/156131026 >> 172.21.15.88:0/156131026 conn(0x562f3f7f6630 msgr2=0x562f3f7fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:06.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.816+0000 7f98c7778b80 1 -- 172.21.15.88:0/156131026 shutdown_connections 2023-07-14T17:44:06.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.816+0000 7f98c7778b80 1 -- 172.21.15.88:0/156131026 wait complete. 2023-07-14T17:44:06.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.816+0000 7f98c7778b80 20 client.15249 trim_cache size 0 max 16384 2023-07-14T17:44:06.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.833+0000 7f98c7778b80 1 Processor -- start 2023-07-14T17:44:06.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.833+0000 7f98c7778b80 1 -- start start 2023-07-14T17:44:06.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f98c7778b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f800f10 0x562f3f814ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f98c7778b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 0x562f3f811dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f98c7778b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f802460 0x562f3f8087f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f98c7778b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f3f6ab4d0 con 0x562f3f802460 2023-07-14T17:44:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f98c7778b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f3f7207c0 con 0x562f3f801860 2023-07-14T17:44:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f98c7778b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f3f710480 con 0x562f3f800f10 2023-07-14T17:44:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f98b9b7d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f802460 0x562f3f8087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f98b9b7d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f802460 0x562f3f8087f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42778/0 (socket says 172.21.15.88:42778) 2023-07-14T17:44:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f98b9b7d700 1 -- 172.21.15.88:0/821416635 learned_addr learned my addr 172.21.15.88:0/821416635 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:06.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f98b8b7b700 1 --2- 172.21.15.88:0/821416635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 0x562f3f811dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f98b937c700 1 --2- 172.21.15.88:0/821416635 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f800f10 0x562f3f814ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f98b9b7d700 1 -- 172.21.15.88:0/821416635 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f800f10 msgr2=0x562f3f814ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f98b9b7d700 1 --2- 172.21.15.88:0/821416635 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f800f10 0x562f3f814ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f98b9b7d700 1 -- 172.21.15.88:0/821416635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 msgr2=0x562f3f811dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f98b9b7d700 1 --2- 172.21.15.88:0/821416635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 0x562f3f811dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f98b9b7d700 1 -- 172.21.15.88:0/821416635 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f3f80ba90 con 0x562f3f802460 2023-07-14T17:44:06.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f98b9b7d700 1 --2- 172.21.15.88:0/821416635 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f802460 0x562f3f8087f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00b0a0 tx=0x7f98ac009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=345b0c45b99a173d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f989ffff700 1 -- 172.21.15.88:0/821416635 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac00e040 con 0x562f3f802460 2023-07-14T17:44:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f989ffff700 1 -- 172.21.15.88:0/821416635 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f98ac020070 con 0x562f3f802460 2023-07-14T17:44:06.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.834+0000 7f989ffff700 1 -- 172.21.15.88:0/821416635 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac004cb0 con 0x562f3f802460 2023-07-14T17:44:06.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.835+0000 7f98c7778b80 1 -- 172.21.15.88:0/821416635 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f802460 msgr2=0x562f3f8087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.835+0000 7f98c7778b80 1 --2- 172.21.15.88:0/821416635 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f802460 0x562f3f8087f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f98ac00b0a0 tx=0x7f98ac009390 comp rx=0 tx=0).stop 2023-07-14T17:44:06.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.835+0000 7f98c7778b80 1 -- 172.21.15.88:0/821416635 shutdown_connections 2023-07-14T17:44:06.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.835+0000 7f98c7778b80 1 --2- 172.21.15.88:0/821416635 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f802460 0x562f3f8087f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.835+0000 7f98c7778b80 1 --2- 172.21.15.88:0/821416635 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f800f10 0x562f3f814ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.835+0000 7f98c7778b80 1 --2- 172.21.15.88:0/821416635 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 0x562f3f811dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.835+0000 7f98c7778b80 1 -- 172.21.15.88:0/821416635 >> 172.21.15.88:0/821416635 conn(0x562f3f7f6630 msgr2=0x562f3f7f7b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:06.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.835+0000 7f98c7778b80 1 -- 172.21.15.88:0/821416635 shutdown_connections 2023-07-14T17:44:06.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.835+0000 7f98c7778b80 1 -- 172.21.15.88:0/821416635 wait complete. 2023-07-14T17:44:06.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.835+0000 7f98c7778b80 1 -- start start 2023-07-14T17:44:06.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.835+0000 7f98c7778b80 1 Processor -- start 2023-07-14T17:44:06.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98c7778b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 0x562f3f7fbb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98c7778b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 0x562f3f811dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98c7778b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f802460 0x562f3f8697b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98c7778b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f3f869ec0 con 0x562f3f800f10 2023-07-14T17:44:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98c7778b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f3f86a030 con 0x562f3f801860 2023-07-14T17:44:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98c7778b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f3f86a1f0 con 0x562f3f802460 2023-07-14T17:44:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98c7778b80 10 client.0 _ll_register_callbacks cb 0x7f98b0095f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98b937c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 0x562f3f7fbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98b937c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 0x562f3f7fbb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42780/0 (socket says 172.21.15.88:42780) 2023-07-14T17:44:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98b937c700 1 -- 172.21.15.88:0/1217753145 learned_addr learned my addr 172.21.15.88:0/1217753145 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98b9b7d700 1 --2- 172.21.15.88:0/1217753145 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f802460 0x562f3f8697b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98b8b7b700 1 --2- 172.21.15.88:0/1217753145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 0x562f3f811dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98b937c700 1 -- 172.21.15.88:0/1217753145 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f802460 msgr2=0x562f3f8697b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98b937c700 1 --2- 172.21.15.88:0/1217753145 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f802460 0x562f3f8697b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98b937c700 1 -- 172.21.15.88:0/1217753145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 msgr2=0x562f3f811dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98b937c700 1 --2- 172.21.15.88:0/1217753145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 0x562f3f811dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98b937c700 1 -- 172.21.15.88:0/1217753145 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b4107880 con 0x562f3f800f10 2023-07-14T17:44:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98b8b7b700 1 --2- 172.21.15.88:0/1217753145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 0x562f3f811dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:06.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98b937c700 1 --2- 172.21.15.88:0/1217753145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 0x562f3f7fbb40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f98a800e4e0 tx=0x7f98a801d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e2652ebb396481 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f989cff9700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f989cff9700 1 -- 172.21.15.88:0/1217753145 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a801ce80 con 0x562f3f800f10 2023-07-14T17:44:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f989cff9700 1 -- 172.21.15.88:0/1217753145 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f98a80056a0 con 0x562f3f800f10 2023-07-14T17:44:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.836+0000 7f98c7778b80 1 -- 172.21.15.88:0/1217753145 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f98a8021070 con 0x562f3f800f10 2023-07-14T17:44:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.837+0000 7f989cff9700 1 -- 172.21.15.88:0/1217753145 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98a801c410 con 0x562f3f800f10 2023-07-14T17:44:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.837+0000 7f98c7778b80 10 client.15255 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.837+0000 7f98c7778b80 1 -- 172.21.15.88:0/1217753145 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x562f3f568c10 con 0x562f3f800f10 2023-07-14T17:44:06.827 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.837+0000 7f989cff9700 1 -- 172.21.15.88:0/1217753145 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f98a8046020 con 0x562f3f800f10 2023-07-14T17:44:06.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.837+0000 7f989cff9700 1 -- 172.21.15.88:0/1217753145 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f98a800e7f0 con 0x562f3f800f10 2023-07-14T17:44:06.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.837+0000 7f98c7778b80 10 client.15255 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:06.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.837+0000 7f98c7778b80 10 client.15255 resolve_mds: resolved name 'c' to daemon mds.c 2023-07-14T17:44:06.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.837+0000 7f98c7778b80 20 client.15255 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:06.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.837+0000 7f98c7778b80 1 --2- 172.21.15.88:0/1217753145 >> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] conn(0x562f3f86ecd0 0x562f3f86b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.828 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.837+0000 7f98c7778b80 4 client.15255 mds_command: new command op to 24454 tid=0[{"prefix": "status"}] 2023-07-14T17:44:06.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.837+0000 7f98c7778b80 1 -- 172.21.15.88:0/1217753145 --> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562f3f74ae60 con 0x562f3f86ecd0 2023-07-14T17:44:06.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.837+0000 7f98b8b7b700 1 --2- 172.21.15.88:0/1217753145 >> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] conn(0x562f3f86ecd0 0x562f3f86b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.838+0000 7f989cff9700 10 client.15255 ms_handle_connect on v2:172.21.15.178:6836/3950404084 2023-07-14T17:44:06.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.838+0000 7f98b8b7b700 1 --2- 172.21.15.88:0/1217753145 >> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] conn(0x562f3f86ecd0 0x562f3f86b5e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=866b6cbe53b4fe58 server_cookie=3f2cdd9a385ee243 in_seq=0 out_seq=0 2023-07-14T17:44:06.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.838+0000 7f989cff9700 1 -- 172.21.15.88:0/1217753145 <== mds.3 v2:172.21.15.178:6836/3950404084 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x562f3f74ae60 con 0x562f3f86ecd0 2023-07-14T17:44:06.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.838+0000 7f989cff9700 10 client.15255 handle_command_reply: tid=0 2023-07-14T17:44:06.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.838+0000 7f98c7778b80 1 client.15255 shutdown 2023-07-14T17:44:06.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.838+0000 7f98c7778b80 1 -- 172.21.15.88:0/1217753145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 msgr2=0x562f3f7fbb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.838+0000 7f98c7778b80 1 --2- 172.21.15.88:0/1217753145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 0x562f3f7fbb40 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f98a800e4e0 tx=0x7f98a801d490 comp rx=0 tx=0).stop 2023-07-14T17:44:06.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.838+0000 7f98c7778b80 1 -- 172.21.15.88:0/1217753145 shutdown_connections 2023-07-14T17:44:06.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.838+0000 7f98c7778b80 1 --2- 172.21.15.88:0/1217753145 >> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] conn(0x562f3f86ecd0 0x562f3f86b5e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.838+0000 7f98c7778b80 1 --2- 172.21.15.88:0/1217753145 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562f3f802460 0x562f3f8697b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.838+0000 7f98c7778b80 1 --2- 172.21.15.88:0/1217753145 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562f3f800f10 0x562f3f7fbb40 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.838+0000 7f98c7778b80 1 --2- 172.21.15.88:0/1217753145 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562f3f801860 0x562f3f811dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.838+0000 7f98c7778b80 1 -- 172.21.15.88:0/1217753145 >> 172.21.15.88:0/1217753145 conn(0x562f3f7f6630 msgr2=0x562f3f814ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:06.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.838+0000 7f989cff9700 0 client.15255 ms_handle_reset on v2:172.21.15.178:6836/3950404084 2023-07-14T17:44:06.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.838+0000 7f98c7778b80 1 -- 172.21.15.88:0/1217753145 shutdown_connections 2023-07-14T17:44:06.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.838+0000 7f98c7778b80 1 -- 172.21.15.88:0/1217753145 wait complete. 2023-07-14T17:44:06.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.838+0000 7f98c7778b80 20 client.15255 trim_cache size 0 max 16384 2023-07-14T17:44:06.831 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-07-14T17:44:06.832 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "b462a556-226c-11ee-9b33-001a4aab830c", 2023-07-14T17:44:06.832 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 3, 2023-07-14T17:44:06.832 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 24454, 2023-07-14T17:44:06.832 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2023-07-14T17:44:06.832 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2023-07-14T17:44:06.832 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2023-07-14T17:44:06.832 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 64.085781999999995, 2023-07-14T17:44:06.833 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 27, 2023-07-14T17:44:06.833 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 88, 2023-07-14T17:44:06.833 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 86, 2023-07-14T17:44:06.833 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 100.468111276 2023-07-14T17:44:06.833 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-07-14T17:44:06.834 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.839+0000 7f98bb5e0700 1 -- 172.21.15.88:0/4271950960 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f98a00a11e0 msgr2=0x7f98a00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.839+0000 7f98bb5e0700 1 --2- 172.21.15.88:0/4271950960 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f98a00a11e0 0x7f98a00a3690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f98a8009c60 tx=0x7f98a8009260 comp rx=0 tx=0).stop 2023-07-14T17:44:06.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.839+0000 7f98bb5e0700 1 -- 172.21.15.88:0/4271950960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98b40ffdc0 msgr2=0x7f98b419a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.839+0000 7f98bb5e0700 1 --2- 172.21.15.88:0/4271950960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98b40ffdc0 0x7f98b419a6a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f98b000d980 tx=0x7f98b000dc90 comp rx=0 tx=0).stop 2023-07-14T17:44:06.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.839+0000 7f98bb5e0700 1 -- 172.21.15.88:0/4271950960 shutdown_connections 2023-07-14T17:44:06.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.839+0000 7f98bb5e0700 1 --2- 172.21.15.88:0/4271950960 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f98b4100770 0x7f98b419ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.835 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.839+0000 7f98bb5e0700 1 --2- 172.21.15.88:0/4271950960 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f98b40feb30 0x7f98b40694e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.839+0000 7f98bb5e0700 1 --2- 172.21.15.88:0/4271950960 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f98a00a11e0 0x7f98a00a3690 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.839+0000 7f98bb5e0700 1 --2- 172.21.15.88:0/4271950960 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98b40ffdc0 0x7f98b419a6a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.839+0000 7f98bb5e0700 1 -- 172.21.15.88:0/4271950960 >> 172.21.15.88:0/4271950960 conn(0x7f98b40fa150 msgr2=0x7f98b40fc410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:06.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.839+0000 7f98bb5e0700 1 -- 172.21.15.88:0/4271950960 shutdown_connections 2023-07-14T17:44:06.836 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.839+0000 7f98bb5e0700 1 -- 172.21.15.88:0/4271950960 wait complete. 2023-07-14T17:44:06.837 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-07-14T17:44:06.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.978+0000 7f00cc564700 1 -- 172.21.15.88:0/4243497498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c40ffdc0 msgr2=0x7f00c4100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.978+0000 7f00cc564700 1 --2- 172.21.15.88:0/4243497498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c40ffdc0 0x7f00c4100230 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f00b4009990 tx=0x7f00b4009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.979+0000 7f00cc564700 1 -- 172.21.15.88:0/4243497498 shutdown_connections 2023-07-14T17:44:06.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.979+0000 7f00cc564700 1 --2- 172.21.15.88:0/4243497498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00c4100770 0x7f00c4107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.979+0000 7f00cc564700 1 --2- 172.21.15.88:0/4243497498 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f00c40feb30 0x7f00c40fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.979+0000 7f00cc564700 1 --2- 172.21.15.88:0/4243497498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c40ffdc0 0x7f00c4100230 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.979+0000 7f00cc564700 1 -- 172.21.15.88:0/4243497498 >> 172.21.15.88:0/4243497498 conn(0x7f00c40fa150 msgr2=0x7f00c40fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:06.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.979+0000 7f00cc564700 1 -- 172.21.15.88:0/4243497498 shutdown_connections 2023-07-14T17:44:06.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.979+0000 7f00cc564700 1 -- 172.21.15.88:0/4243497498 wait complete. 2023-07-14T17:44:06.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.979+0000 7f00cc564700 1 Processor -- start 2023-07-14T17:44:06.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.979+0000 7f00cc564700 1 -- start start 2023-07-14T17:44:06.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.979+0000 7f00cc564700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c40feb30 0x7f00c4194770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.979+0000 7f00cc564700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f00c40ffdc0 0x7f00c4194cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.979+0000 7f00cc564700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00c4100770 0x7f00c419bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.979+0000 7f00cc564700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c419c2f0 con 0x7f00c40feb30 2023-07-14T17:44:06.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.979+0000 7f00cc564700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c419c430 con 0x7f00c4100770 2023-07-14T17:44:06.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.979+0000 7f00cc564700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c419c5c0 con 0x7f00c40ffdc0 2023-07-14T17:44:06.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.979+0000 7f00ca300700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c40feb30 0x7f00c4194770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.979+0000 7f00ca300700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c40feb30 0x7f00c4194770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42794/0 (socket says 172.21.15.88:42794) 2023-07-14T17:44:06.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.979+0000 7f00ca300700 1 -- 172.21.15.88:0/3678154555 learned_addr learned my addr 172.21.15.88:0/3678154555 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:06.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.980+0000 7f00cab01700 1 --2- 172.21.15.88:0/3678154555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00c4100770 0x7f00c419bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.980+0000 7f00c9aff700 1 --2- 172.21.15.88:0/3678154555 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f00c40ffdc0 0x7f00c4194cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.980+0000 7f00ca300700 1 -- 172.21.15.88:0/3678154555 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f00c40ffdc0 msgr2=0x7f00c4194cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.980+0000 7f00ca300700 1 --2- 172.21.15.88:0/3678154555 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f00c40ffdc0 0x7f00c4194cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.980+0000 7f00ca300700 1 -- 172.21.15.88:0/3678154555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00c4100770 msgr2=0x7f00c419bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.965 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.980+0000 7f00ca300700 1 --2- 172.21.15.88:0/3678154555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00c4100770 0x7f00c419bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.980+0000 7f00ca300700 1 -- 172.21.15.88:0/3678154555 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b40095f0 con 0x7f00c40feb30 2023-07-14T17:44:06.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.980+0000 7f00c9aff700 1 --2- 172.21.15.88:0/3678154555 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f00c40ffdc0 0x7f00c4194cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:06.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.980+0000 7f00ca300700 1 --2- 172.21.15.88:0/3678154555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c40feb30 0x7f00c4194770 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f00c000c920 tx=0x7f00c000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.966 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.980+0000 7f00bb7fe700 1 -- 172.21.15.88:0/3678154555 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c0007b00 con 0x7f00c40feb30 2023-07-14T17:44:06.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.980+0000 7f00cc564700 1 -- 172.21.15.88:0/3678154555 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c419c8a0 con 0x7f00c40feb30 2023-07-14T17:44:06.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.980+0000 7f00bb7fe700 1 -- 172.21.15.88:0/3678154555 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f00c000ce70 con 0x7f00c40feb30 2023-07-14T17:44:06.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.980+0000 7f00bb7fe700 1 -- 172.21.15.88:0/3678154555 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c0005440 con 0x7f00c40feb30 2023-07-14T17:44:06.967 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.980+0000 7f00cc564700 1 -- 172.21.15.88:0/3678154555 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c419cdb0 con 0x7f00c40feb30 2023-07-14T17:44:06.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.982+0000 7f00bb7fe700 1 -- 172.21.15.88:0/3678154555 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f00c00055a0 con 0x7f00c40feb30 2023-07-14T17:44:06.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.983+0000 7f00bb7fe700 1 --2- 172.21.15.88:0/3678154555 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f00b00a11e0 0x7f00b00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.983+0000 7f00c9aff700 1 --2- 172.21.15.88:0/3678154555 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f00b00a11e0 0x7f00b00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.983+0000 7f00bb7fe700 1 -- 172.21.15.88:0/3678154555 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f00c00cc9e0 con 0x7f00c40feb30 2023-07-14T17:44:06.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.983+0000 7f00c9aff700 1 --2- 172.21.15.88:0/3678154555 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f00b00a11e0 0x7f00b00a3690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f00b40095c0 tx=0x7f00b4009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.974 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00d86fcb80 1 Processor -- start 2023-07-14T17:44:06.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00d86fcb80 1 -- start start 2023-07-14T17:44:06.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00d86fcb80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e0900d0 0x55a41e0904e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00d86fcb80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e090a20 0x55a41e0910c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.975 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00d86fcb80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e091650 0x55a41e091ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00d86fcb80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a41df92c60 con 0x55a41e0900d0 2023-07-14T17:44:06.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00d86fcb80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a41dfe59f0 con 0x55a41e090a20 2023-07-14T17:44:06.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00d86fcb80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a41dfdee80 con 0x55a41e091650 2023-07-14T17:44:06.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00ca300700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e0900d0 0x55a41e0904e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00ca300700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e0900d0 0x55a41e0904e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42802/0 (socket says 172.21.15.88:42802) 2023-07-14T17:44:06.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00ca300700 1 -- 172.21.15.88:0/1434360907 learned_addr learned my addr 172.21.15.88:0/1434360907 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:06.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00c9aff700 1 --2- 172.21.15.88:0/1434360907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e090a20 0x55a41e0910c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00ca300700 1 -- 172.21.15.88:0/1434360907 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e091650 msgr2=0x55a41e091ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00cab01700 1 --2- 172.21.15.88:0/1434360907 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e091650 0x55a41e091ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00ca300700 1 --2- 172.21.15.88:0/1434360907 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e091650 0x55a41e091ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00ca300700 1 -- 172.21.15.88:0/1434360907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e090a20 msgr2=0x55a41e0910c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00ca300700 1 --2- 172.21.15.88:0/1434360907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e090a20 0x55a41e0910c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00ca300700 1 -- 172.21.15.88:0/1434360907 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a41de0cd90 con 0x55a41e0900d0 2023-07-14T17:44:06.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00cab01700 1 --2- 172.21.15.88:0/1434360907 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e091650 0x55a41e091ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:06.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.984+0000 7f00c9aff700 1 --2- 172.21.15.88:0/1434360907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e090a20 0x55a41e0910c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:06.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.985+0000 7f00ca300700 1 --2- 172.21.15.88:0/1434360907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e0900d0 0x55a41e0904e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f00c00998b0 tx=0x7f00c00993b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4857c528d3440802 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.985+0000 7f00b8ff9700 1 -- 172.21.15.88:0/1434360907 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c00a1d70 con 0x55a41e0900d0 2023-07-14T17:44:06.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.985+0000 7f00b8ff9700 1 -- 172.21.15.88:0/1434360907 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f00c00ca270 con 0x55a41e0900d0 2023-07-14T17:44:06.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.985+0000 7f00b8ff9700 1 -- 172.21.15.88:0/1434360907 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c0099010 con 0x55a41e0900d0 2023-07-14T17:44:06.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.985+0000 7f00d86fcb80 1 -- 172.21.15.88:0/1434360907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e0900d0 msgr2=0x55a41e0904e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.985+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/1434360907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e0900d0 0x55a41e0904e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f00c00998b0 tx=0x7f00c00993b0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.985+0000 7f00d86fcb80 1 -- 172.21.15.88:0/1434360907 shutdown_connections 2023-07-14T17:44:06.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.985+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/1434360907 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e091650 0x55a41e091ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.985+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/1434360907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e0900d0 0x55a41e0904e0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.985+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/1434360907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e090a20 0x55a41e0910c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.985+0000 7f00d86fcb80 1 -- 172.21.15.88:0/1434360907 >> 172.21.15.88:0/1434360907 conn(0x55a41e083630 msgr2=0x55a41e088980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:06.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.986+0000 7f00d86fcb80 1 -- 172.21.15.88:0/1434360907 shutdown_connections 2023-07-14T17:44:06.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.986+0000 7f00d86fcb80 1 -- 172.21.15.88:0/1434360907 wait complete. 2023-07-14T17:44:06.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.986+0000 7f00d86fcb80 1 -- start start 2023-07-14T17:44:06.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.986+0000 7f00d86fcb80 1 Processor -- start 2023-07-14T17:44:06.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.986+0000 7f00d86fcb80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e0900d0 0x55a41e099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.986+0000 7f00d86fcb80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e090a20 0x55a41e099f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.986+0000 7f00d86fcb80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 0x55a41e0a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.986+0000 7f00d86fcb80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a41df7ea70 con 0x55a41e091650 2023-07-14T17:44:06.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.986+0000 7f00d86fcb80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a41df7e770 con 0x55a41e0900d0 2023-07-14T17:44:06.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.986+0000 7f00d86fcb80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a41de628d0 con 0x55a41e090a20 2023-07-14T17:44:06.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.986+0000 7f00d86fcb80 10 client.0 _ll_register_callbacks cb 0x55a41df7c710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:06.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.986+0000 7f00cab01700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 0x55a41e0a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.986+0000 7f00cab01700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 0x55a41e0a0fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42812/0 (socket says 172.21.15.88:42812) 2023-07-14T17:44:06.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.986+0000 7f00cab01700 1 -- 172.21.15.88:0/3649048416 learned_addr learned my addr 172.21.15.88:0/3649048416 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:06.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.986+0000 7f00ca300700 1 --2- 172.21.15.88:0/3649048416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e0900d0 0x55a41e099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00c9aff700 1 --2- 172.21.15.88:0/3649048416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e090a20 0x55a41e099f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.985 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00cab01700 1 -- 172.21.15.88:0/3649048416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e090a20 msgr2=0x55a41e099f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00cab01700 1 --2- 172.21.15.88:0/3649048416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e090a20 0x55a41e099f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00cab01700 1 -- 172.21.15.88:0/3649048416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e0900d0 msgr2=0x55a41e099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00cab01700 1 --2- 172.21.15.88:0/3649048416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e0900d0 0x55a41e099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00cab01700 1 -- 172.21.15.88:0/3649048416 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c419cdb0 con 0x55a41e091650 2023-07-14T17:44:06.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00c9aff700 1 --2- 172.21.15.88:0/3649048416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e090a20 0x55a41e099f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:06.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00ca300700 1 --2- 172.21.15.88:0/3649048416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e0900d0 0x55a41e099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T17:44:06.986 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00cab01700 1 --2- 172.21.15.88:0/3649048416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 0x55a41e0a0fd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f00bc004040 tx=0x7f00bc004350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9663c3a68632a5e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00adffb700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:06.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00adffb700 1 -- 172.21.15.88:0/3649048416 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc004bb0 con 0x55a41e091650 2023-07-14T17:44:06.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00adffb700 1 -- 172.21.15.88:0/3649048416 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f00bc004d10 con 0x55a41e091650 2023-07-14T17:44:06.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00d86fcb80 1 -- 172.21.15.88:0/3649048416 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a41df384a0 con 0x55a41e091650 2023-07-14T17:44:06.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00adffb700 1 -- 172.21.15.88:0/3649048416 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00bc007710 con 0x55a41e091650 2023-07-14T17:44:06.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00d86fcb80 10 client.15267 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:06.987 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00d86fcb80 1 -- 172.21.15.88:0/3649048416 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a41ddf5c10 con 0x55a41e091650 2023-07-14T17:44:06.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00adffb700 1 -- 172.21.15.88:0/3649048416 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55a41df384a0 con 0x55a41e091650 2023-07-14T17:44:06.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00adffb700 1 -- 172.21.15.88:0/3649048416 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f00bc014070 con 0x55a41e091650 2023-07-14T17:44:06.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.987+0000 7f00d86fcb80 10 client.15267 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:06.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.988+0000 7f00d86fcb80 10 client.15267 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T17:44:06.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.988+0000 7f00d86fcb80 20 client.15267 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:06.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.988+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/3649048416 >> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] conn(0x55a41e0a4cf0 0x55a41e0a40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.988+0000 7f00d86fcb80 4 client.15267 mds_command: new command op to 24469 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T17:44:06.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.988+0000 7f00d86fcb80 1 -- 172.21.15.88:0/3649048416 --> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a41e05cd50 con 0x55a41e0a4cf0 2023-07-14T17:44:06.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.988+0000 7f00ca300700 1 --2- 172.21.15.88:0/3649048416 >> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] conn(0x55a41e0a4cf0 0x55a41e0a40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.988+0000 7f00adffb700 10 client.15267 ms_handle_connect on v2:172.21.15.178:6838/2953415077 2023-07-14T17:44:06.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.988+0000 7f00ca300700 1 --2- 172.21.15.88:0/3649048416 >> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] conn(0x55a41e0a4cf0 0x55a41e0a40a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d74e4d124e162c30 server_cookie=fb9d5548be22d655 in_seq=0 out_seq=0 2023-07-14T17:44:06.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.989+0000 7f00adffb700 1 -- 172.21.15.88:0/3649048416 <== mds.2 v2:172.21.15.178:6838/2953415077 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55a41e05cd50 con 0x55a41e0a4cf0 2023-07-14T17:44:06.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.989+0000 7f00adffb700 10 client.15267 handle_command_reply: tid=0 2023-07-14T17:44:06.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.989+0000 7f00d86fcb80 1 client.15267 shutdown 2023-07-14T17:44:06.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.989+0000 7f00d86fcb80 1 -- 172.21.15.88:0/3649048416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 msgr2=0x55a41e0a0fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:06.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.989+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/3649048416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 0x55a41e0a0fd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f00bc004040 tx=0x7f00bc004350 comp rx=0 tx=0).stop 2023-07-14T17:44:06.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.989+0000 7f00d86fcb80 1 -- 172.21.15.88:0/3649048416 shutdown_connections 2023-07-14T17:44:06.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.989+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/3649048416 >> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] conn(0x55a41e0a4cf0 0x55a41e0a40a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.989+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/3649048416 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 0x55a41e0a0fd0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.990 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.989+0000 7f00adffb700 0 client.15267 ms_handle_reset on v2:172.21.15.178:6838/2953415077 2023-07-14T17:44:06.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.989+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/3649048416 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e0900d0 0x55a41e099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.989+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/3649048416 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e090a20 0x55a41e099f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.989+0000 7f00d86fcb80 1 -- 172.21.15.88:0/3649048416 >> 172.21.15.88:0/3649048416 conn(0x55a41e083630 msgr2=0x55a41e087600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:06.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.989+0000 7f00d86fcb80 1 -- 172.21.15.88:0/3649048416 shutdown_connections 2023-07-14T17:44:06.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.989+0000 7f00d86fcb80 1 -- 172.21.15.88:0/3649048416 wait complete. 2023-07-14T17:44:06.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:06.989+0000 7f00d86fcb80 20 client.15267 trim_cache size 0 max 16384 2023-07-14T17:44:06.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00d86fcb80 1 Processor -- start 2023-07-14T17:44:06.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00d86fcb80 1 -- start start 2023-07-14T17:44:06.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00d86fcb80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e0900d0 0x55a41e0880a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00d86fcb80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e090a20 0x55a41e0ada60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00d86fcb80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 0x55a41e097a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:06.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00d86fcb80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a41e089800 con 0x55a41e091650 2023-07-14T17:44:06.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00d86fcb80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a41dfad790 con 0x55a41e0900d0 2023-07-14T17:44:06.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00d86fcb80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a41de6ede0 con 0x55a41e090a20 2023-07-14T17:44:06.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00cab01700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 0x55a41e097a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00cab01700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 0x55a41e097a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42820/0 (socket says 172.21.15.88:42820) 2023-07-14T17:44:06.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00cab01700 1 -- 172.21.15.88:0/68962287 learned_addr learned my addr 172.21.15.88:0/68962287 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:06.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00c9aff700 1 --2- 172.21.15.88:0/68962287 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e090a20 0x55a41e0ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00cab01700 1 -- 172.21.15.88:0/68962287 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e090a20 msgr2=0x55a41e0ada60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00cab01700 1 --2- 172.21.15.88:0/68962287 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e090a20 0x55a41e0ada60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00ca300700 1 --2- 172.21.15.88:0/68962287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e0900d0 0x55a41e0880a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:06.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00cab01700 1 -- 172.21.15.88:0/68962287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e0900d0 msgr2=0x55a41e0880a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:06.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00cab01700 1 --2- 172.21.15.88:0/68962287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e0900d0 0x55a41e0880a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:06.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00cab01700 1 -- 172.21.15.88:0/68962287 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a41de33200 con 0x55a41e091650 2023-07-14T17:44:06.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00c9aff700 1 --2- 172.21.15.88:0/68962287 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e090a20 0x55a41e0ada60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:06.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.007+0000 7f00cab01700 1 --2- 172.21.15.88:0/68962287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 0x55a41e097a30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f00bc002230 tx=0x7f00bc0095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7494a409c2ec145 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:06.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.008+0000 7f00b97fa700 1 -- 172.21.15.88:0/68962287 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc01e400 con 0x55a41e091650 2023-07-14T17:44:06.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.008+0000 7f00b97fa700 1 -- 172.21.15.88:0/68962287 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f00bc00f070 con 0x55a41e091650 2023-07-14T17:44:06.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.008+0000 7f00b97fa700 1 -- 172.21.15.88:0/68962287 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc007e50 con 0x55a41e091650 2023-07-14T17:44:07.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.008+0000 7f00d86fcb80 1 -- 172.21.15.88:0/68962287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 msgr2=0x55a41e097a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.008+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/68962287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 0x55a41e097a30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f00bc002230 tx=0x7f00bc0095e0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.008+0000 7f00d86fcb80 1 -- 172.21.15.88:0/68962287 shutdown_connections 2023-07-14T17:44:07.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.008+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/68962287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 0x55a41e097a30 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.008+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/68962287 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e0900d0 0x55a41e0880a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.008+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/68962287 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e090a20 0x55a41e0ada60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.008+0000 7f00d86fcb80 1 -- 172.21.15.88:0/68962287 >> 172.21.15.88:0/68962287 conn(0x55a41e083630 msgr2=0x55a41e098720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:07.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.008+0000 7f00d86fcb80 1 -- 172.21.15.88:0/68962287 shutdown_connections 2023-07-14T17:44:07.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.008+0000 7f00d86fcb80 1 -- 172.21.15.88:0/68962287 wait complete. 2023-07-14T17:44:07.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.008+0000 7f00d86fcb80 1 -- start start 2023-07-14T17:44:07.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00d86fcb80 1 Processor -- start 2023-07-14T17:44:07.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00d86fcb80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e0900d0 0x55a41e0a8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00d86fcb80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e090a20 0x55a41e0a9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00d86fcb80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 0x55a41e0f8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00d86fcb80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a41e0adc50 con 0x55a41e091650 2023-07-14T17:44:07.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00d86fcb80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a41e0addc0 con 0x55a41e0900d0 2023-07-14T17:44:07.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00d86fcb80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a41e0f8eb0 con 0x55a41e090a20 2023-07-14T17:44:07.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00d86fcb80 10 client.0 _ll_register_callbacks cb 0x7f00bc0098c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:07.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00cab01700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 0x55a41e0f8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00cab01700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 0x55a41e0f8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42826/0 (socket says 172.21.15.88:42826) 2023-07-14T17:44:07.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00cab01700 1 -- 172.21.15.88:0/2764871149 learned_addr learned my addr 172.21.15.88:0/2764871149 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00ca300700 1 --2- 172.21.15.88:0/2764871149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e0900d0 0x55a41e0a8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00c9aff700 1 --2- 172.21.15.88:0/2764871149 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e090a20 0x55a41e0a9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00cab01700 1 -- 172.21.15.88:0/2764871149 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e090a20 msgr2=0x55a41e0a9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00cab01700 1 --2- 172.21.15.88:0/2764871149 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e090a20 0x55a41e0a9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00cab01700 1 -- 172.21.15.88:0/2764871149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e0900d0 msgr2=0x55a41e0a8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00cab01700 1 --2- 172.21.15.88:0/2764871149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e0900d0 0x55a41e0a8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00cab01700 1 -- 172.21.15.88:0/2764871149 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a41ddf5c10 con 0x55a41e091650 2023-07-14T17:44:07.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00cab01700 1 --2- 172.21.15.88:0/2764871149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 0x55a41e0f8970 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f00bc0093f0 tx=0x7f00bc004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=759bc8e636d7c703 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:07.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00adffb700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:07.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00adffb700 1 -- 172.21.15.88:0/2764871149 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc007920 con 0x55a41e091650 2023-07-14T17:44:07.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00adffb700 1 -- 172.21.15.88:0/2764871149 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f00bc01e400 con 0x55a41e091650 2023-07-14T17:44:07.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.009+0000 7f00d86fcb80 1 -- 172.21.15.88:0/2764871149 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f00c00ca270 con 0x55a41e091650 2023-07-14T17:44:07.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.010+0000 7f00adffb700 1 -- 172.21.15.88:0/2764871149 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f00bc01e5c0 con 0x55a41e091650 2023-07-14T17:44:07.004 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.010+0000 7f00d86fcb80 10 client.15273 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:07.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.010+0000 7f00d86fcb80 1 -- 172.21.15.88:0/2764871149 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a41de0b260 con 0x55a41e091650 2023-07-14T17:44:07.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.010+0000 7f00adffb700 1 -- 172.21.15.88:0/2764871149 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f00c00ca270 con 0x55a41e091650 2023-07-14T17:44:07.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.010+0000 7f00adffb700 1 -- 172.21.15.88:0/2764871149 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f00bc014050 con 0x55a41e091650 2023-07-14T17:44:07.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.010+0000 7f00d86fcb80 10 client.15273 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:07.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.010+0000 7f00d86fcb80 10 client.15273 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T17:44:07.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.010+0000 7f00d86fcb80 20 client.15273 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:07.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.010+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/2764871149 >> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] conn(0x55a41e0fd7d0 0x55a41e0fa1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.010+0000 7f00d86fcb80 4 client.15273 mds_command: new command op to 24469 tid=0[{"prefix": "status"}] 2023-07-14T17:44:07.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.010+0000 7f00d86fcb80 1 -- 172.21.15.88:0/2764871149 --> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a41dfd7e60 con 0x55a41e0fd7d0 2023-07-14T17:44:07.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.010+0000 7f00ca300700 1 --2- 172.21.15.88:0/2764871149 >> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] conn(0x55a41e0fd7d0 0x55a41e0fa1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.011+0000 7f00adffb700 10 client.15273 ms_handle_connect on v2:172.21.15.178:6838/2953415077 2023-07-14T17:44:07.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.011+0000 7f00ca300700 1 --2- 172.21.15.88:0/2764871149 >> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] conn(0x55a41e0fd7d0 0x55a41e0fa1b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=948e3d48ba7c7607 server_cookie=a6ea9c37cdc446f2 in_seq=0 out_seq=0 2023-07-14T17:44:07.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.011+0000 7f00adffb700 1 -- 172.21.15.88:0/2764871149 <== mds.2 v2:172.21.15.178:6838/2953415077 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55a41dfd7e60 con 0x55a41e0fd7d0 2023-07-14T17:44:07.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.011+0000 7f00adffb700 10 client.15273 handle_command_reply: tid=0 2023-07-14T17:44:07.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.011+0000 7f00d86fcb80 1 client.15273 shutdown 2023-07-14T17:44:07.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.011+0000 7f00d86fcb80 1 -- 172.21.15.88:0/2764871149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 msgr2=0x55a41e0f8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.011+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/2764871149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 0x55a41e0f8970 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f00bc0093f0 tx=0x7f00bc004860 comp rx=0 tx=0).stop 2023-07-14T17:44:07.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.011+0000 7f00d86fcb80 1 -- 172.21.15.88:0/2764871149 shutdown_connections 2023-07-14T17:44:07.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.011+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/2764871149 >> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] conn(0x55a41e0fd7d0 0x55a41e0fa1b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.011+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/2764871149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55a41e091650 0x55a41e0f8970 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.011+0000 7f00adffb700 0 client.15273 ms_handle_reset on v2:172.21.15.178:6838/2953415077 2023-07-14T17:44:07.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.011+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/2764871149 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55a41e0900d0 0x55a41e0a8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.011+0000 7f00d86fcb80 1 --2- 172.21.15.88:0/2764871149 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55a41e090a20 0x55a41e0a9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.011+0000 7f00d86fcb80 1 -- 172.21.15.88:0/2764871149 >> 172.21.15.88:0/2764871149 conn(0x55a41e083630 msgr2=0x55a41e097a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:07.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.011+0000 7f00d86fcb80 1 -- 172.21.15.88:0/2764871149 shutdown_connections 2023-07-14T17:44:07.008 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.011+0000 7f00d86fcb80 1 -- 172.21.15.88:0/2764871149 wait complete. 2023-07-14T17:44:07.009 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.011+0000 7f00d86fcb80 20 client.15273 trim_cache size 0 max 16384 2023-07-14T17:44:07.009 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-07-14T17:44:07.009 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "b462a556-226c-11ee-9b33-001a4aab830c", 2023-07-14T17:44:07.009 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 2, 2023-07-14T17:44:07.010 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 24469, 2023-07-14T17:44:07.010 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2023-07-14T17:44:07.010 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2023-07-14T17:44:07.010 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2023-07-14T17:44:07.010 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 65.266552168000004, 2023-07-14T17:44:07.010 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 27, 2023-07-14T17:44:07.010 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 88, 2023-07-14T17:44:07.010 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 86, 2023-07-14T17:44:07.011 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 95.979509230000005 2023-07-14T17:44:07.011 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-07-14T17:44:07.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.012+0000 7f00cc564700 1 -- 172.21.15.88:0/3678154555 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f00b00a11e0 msgr2=0x7f00b00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.012+0000 7f00cc564700 1 --2- 172.21.15.88:0/3678154555 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f00b00a11e0 0x7f00b00a3690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f00b40095c0 tx=0x7f00b4009510 comp rx=0 tx=0).stop 2023-07-14T17:44:07.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.012+0000 7f00cc564700 1 -- 172.21.15.88:0/3678154555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c40feb30 msgr2=0x7f00c4194770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.012+0000 7f00cc564700 1 --2- 172.21.15.88:0/3678154555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c40feb30 0x7f00c4194770 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f00c000c920 tx=0x7f00c000cc30 comp rx=0 tx=0).stop 2023-07-14T17:44:07.012 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.012+0000 7f00cc564700 1 -- 172.21.15.88:0/3678154555 shutdown_connections 2023-07-14T17:44:07.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.012+0000 7f00cc564700 1 --2- 172.21.15.88:0/3678154555 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00c4100770 0x7f00c419bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.012+0000 7f00cc564700 1 --2- 172.21.15.88:0/3678154555 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f00b00a11e0 0x7f00b00a3690 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.012+0000 7f00cc564700 1 --2- 172.21.15.88:0/3678154555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00c40feb30 0x7f00c4194770 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.012+0000 7f00cc564700 1 --2- 172.21.15.88:0/3678154555 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f00c40ffdc0 0x7f00c4194cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.012+0000 7f00cc564700 1 -- 172.21.15.88:0/3678154555 >> 172.21.15.88:0/3678154555 conn(0x7f00c40fa150 msgr2=0x7f00c40fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:07.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.012+0000 7f00cc564700 1 -- 172.21.15.88:0/3678154555 shutdown_connections 2023-07-14T17:44:07.013 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.012+0000 7f00cc564700 1 -- 172.21.15.88:0/3678154555 wait complete. 2023-07-14T17:44:07.014 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-07-14T17:44:07.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.154+0000 7ff7717d9700 1 -- 172.21.15.88:0/2176183086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff76c100690 msgr2=0x7ff76c106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.154+0000 7ff7717d9700 1 --2- 172.21.15.88:0/2176183086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff76c100690 0x7ff76c106f50 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7ff76400b0a0 tx=0x7ff76400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.154+0000 7ff7717d9700 1 -- 172.21.15.88:0/2176183086 shutdown_connections 2023-07-14T17:44:07.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.154+0000 7ff7717d9700 1 --2- 172.21.15.88:0/2176183086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff76c100690 0x7ff76c106f50 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.154+0000 7ff7717d9700 1 --2- 172.21.15.88:0/2176183086 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff76c0feb10 0x7ff76c0fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.136 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.154+0000 7ff7717d9700 1 --2- 172.21.15.88:0/2176183086 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff76c0ffd00 0x7ff76c100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.154+0000 7ff7717d9700 1 -- 172.21.15.88:0/2176183086 >> 172.21.15.88:0/2176183086 conn(0x7ff76c0fa130 msgr2=0x7ff76c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:07.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.154+0000 7ff7717d9700 1 -- 172.21.15.88:0/2176183086 shutdown_connections 2023-07-14T17:44:07.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.154+0000 7ff7717d9700 1 -- 172.21.15.88:0/2176183086 wait complete. 2023-07-14T17:44:07.137 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.154+0000 7ff7717d9700 1 Processor -- start 2023-07-14T17:44:07.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.154+0000 7ff7717d9700 1 -- start start 2023-07-14T17:44:07.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff7717d9700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff76c0feb10 0x7ff76c195a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff7717d9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff76c0ffd00 0x7ff76c195f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.138 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff7717d9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff76c100690 0x7ff76c19cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff7717d9700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c19d580 con 0x7ff76c100690 2023-07-14T17:44:07.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff7717d9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c19d6c0 con 0x7ff76c0ffd00 2023-07-14T17:44:07.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff7717d9700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c19d850 con 0x7ff76c0feb10 2023-07-14T17:44:07.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff76b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff76c100690 0x7ff76c19cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff76b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff76c100690 0x7ff76c19cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42840/0 (socket says 172.21.15.88:42840) 2023-07-14T17:44:07.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff76b7fe700 1 -- 172.21.15.88:0/588275649 learned_addr learned my addr 172.21.15.88:0/588275649 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:07.139 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff76affd700 1 --2- 172.21.15.88:0/588275649 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff76c0feb10 0x7ff76c195a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff76a7fc700 1 --2- 172.21.15.88:0/588275649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff76c0ffd00 0x7ff76c195f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff76b7fe700 1 -- 172.21.15.88:0/588275649 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff76c0feb10 msgr2=0x7ff76c195a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff76b7fe700 1 --2- 172.21.15.88:0/588275649 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff76c0feb10 0x7ff76c195a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff76b7fe700 1 -- 172.21.15.88:0/588275649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff76c0ffd00 msgr2=0x7ff76c195f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff76b7fe700 1 --2- 172.21.15.88:0/588275649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff76c0ffd00 0x7ff76c195f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff76b7fe700 1 -- 172.21.15.88:0/588275649 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff764009d00 con 0x7ff76c100690 2023-07-14T17:44:07.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff76affd700 1 --2- 172.21.15.88:0/588275649 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff76c0feb10 0x7ff76c195a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:07.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff76a7fc700 1 --2- 172.21.15.88:0/588275649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff76c0ffd00 0x7ff76c195f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:07.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff76b7fe700 1 --2- 172.21.15.88:0/588275649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff76c100690 0x7ff76c19cfb0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff76400b830 tx=0x7ff764004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:07.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff753fff700 1 -- 172.21.15.88:0/588275649 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76400e040 con 0x7ff76c100690 2023-07-14T17:44:07.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff7717d9700 1 -- 172.21.15.88:0/588275649 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff76c19dad0 con 0x7ff76c100690 2023-07-14T17:44:07.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff753fff700 1 -- 172.21.15.88:0/588275649 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff764009440 con 0x7ff76c100690 2023-07-14T17:44:07.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff753fff700 1 -- 172.21.15.88:0/588275649 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76401dbe0 con 0x7ff76c100690 2023-07-14T17:44:07.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.155+0000 7ff7717d9700 1 -- 172.21.15.88:0/588275649 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff76c19dfd0 con 0x7ff76c100690 2023-07-14T17:44:07.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.158+0000 7ff753fff700 1 -- 172.21.15.88:0/588275649 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff764005450 con 0x7ff76c100690 2023-07-14T17:44:07.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.159+0000 7ff753fff700 1 --2- 172.21.15.88:0/588275649 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff7540a11e0 0x7ff7540a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.159+0000 7ff76affd700 1 --2- 172.21.15.88:0/588275649 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff7540a11e0 0x7ff7540a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.159+0000 7ff753fff700 1 -- 172.21.15.88:0/588275649 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff7640ce850 con 0x7ff76c100690 2023-07-14T17:44:07.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.159+0000 7ff77d971b80 1 Processor -- start 2023-07-14T17:44:07.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.159+0000 7ff76affd700 1 --2- 172.21.15.88:0/588275649 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff7540a11e0 0x7ff7540a3690 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff76c0ffb60 tx=0x7ff760008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:07.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.159+0000 7ff77d971b80 1 -- start start 2023-07-14T17:44:07.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.159+0000 7ff77d971b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a4f10 0x55819a5a5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.159+0000 7ff77d971b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a5860 0x55819a5a5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.159+0000 7ff77d971b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a6460 0x55819a5a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.159+0000 7ff77d971b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55819a4a9c60 con 0x55819a5a5860 2023-07-14T17:44:07.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.159+0000 7ff77d971b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55819a4fc9f0 con 0x55819a5a4f10 2023-07-14T17:44:07.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.159+0000 7ff77d971b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55819a4f5e80 con 0x55819a5a6460 2023-07-14T17:44:07.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.159+0000 7ff76a7fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a5860 0x55819a5a5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.159+0000 7ff76a7fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a5860 0x55819a5a5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42846/0 (socket says 172.21.15.88:42846) 2023-07-14T17:44:07.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.159+0000 7ff76a7fc700 1 -- 172.21.15.88:0/2608825039 learned_addr learned my addr 172.21.15.88:0/2608825039 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:07.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.160+0000 7ff76affd700 1 --2- 172.21.15.88:0/2608825039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a4f10 0x55819a5a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.160+0000 7ff76b7fe700 1 --2- 172.21.15.88:0/2608825039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a6460 0x55819a5a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.160+0000 7ff76a7fc700 1 -- 172.21.15.88:0/2608825039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a6460 msgr2=0x55819a5a68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:07.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.160+0000 7ff76a7fc700 1 --2- 172.21.15.88:0/2608825039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a6460 0x55819a5a68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.160+0000 7ff76a7fc700 1 -- 172.21.15.88:0/2608825039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a4f10 msgr2=0x55819a5a5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:07.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.160+0000 7ff76a7fc700 1 --2- 172.21.15.88:0/2608825039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a4f10 0x55819a5a5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.160+0000 7ff76a7fc700 1 -- 172.21.15.88:0/2608825039 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55819a323d90 con 0x55819a5a5860 2023-07-14T17:44:07.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.160+0000 7ff76b7fe700 1 --2- 172.21.15.88:0/2608825039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a6460 0x55819a5a68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:07.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.160+0000 7ff76a7fc700 1 --2- 172.21.15.88:0/2608825039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a5860 0x55819a5a5e80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009ce0 tx=0x7ff75c00e2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dcf7d68eb28b43c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:07.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.160+0000 7ff7517fa700 1 -- 172.21.15.88:0/2608825039 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c00eb70 con 0x55819a5a5860 2023-07-14T17:44:07.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.160+0000 7ff7517fa700 1 -- 172.21.15.88:0/2608825039 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff75c00ecd0 con 0x55819a5a5860 2023-07-14T17:44:07.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.160+0000 7ff7517fa700 1 -- 172.21.15.88:0/2608825039 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff75c010630 con 0x55819a5a5860 2023-07-14T17:44:07.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.161+0000 7ff77d971b80 1 -- 172.21.15.88:0/2608825039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a5860 msgr2=0x55819a5a5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.161+0000 7ff77d971b80 1 --2- 172.21.15.88:0/2608825039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a5860 0x55819a5a5e80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009ce0 tx=0x7ff75c00e2f0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.161+0000 7ff77d971b80 1 -- 172.21.15.88:0/2608825039 shutdown_connections 2023-07-14T17:44:07.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.161+0000 7ff77d971b80 1 --2- 172.21.15.88:0/2608825039 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a6460 0x55819a5a68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.161+0000 7ff77d971b80 1 --2- 172.21.15.88:0/2608825039 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a4f10 0x55819a5a5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.161+0000 7ff77d971b80 1 --2- 172.21.15.88:0/2608825039 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a5860 0x55819a5a5e80 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.161+0000 7ff77d971b80 1 -- 172.21.15.88:0/2608825039 >> 172.21.15.88:0/2608825039 conn(0x55819a59a630 msgr2=0x55819a59f980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:07.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.161+0000 7ff77d971b80 1 -- 172.21.15.88:0/2608825039 shutdown_connections 2023-07-14T17:44:07.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.161+0000 7ff77d971b80 1 -- 172.21.15.88:0/2608825039 wait complete. 2023-07-14T17:44:07.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.161+0000 7ff77d971b80 1 -- start start 2023-07-14T17:44:07.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.161+0000 7ff77d971b80 1 Processor -- start 2023-07-14T17:44:07.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.161+0000 7ff77d971b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 0x55819a5ae8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.161+0000 7ff77d971b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a5860 0x55819a5aee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.161+0000 7ff77d971b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a6460 0x55819a5b5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.161+0000 7ff77d971b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55819a495a70 con 0x55819a5a4f10 2023-07-14T17:44:07.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.161+0000 7ff77d971b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55819a495770 con 0x55819a5a5860 2023-07-14T17:44:07.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.161+0000 7ff77d971b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55819a3798d0 con 0x55819a5a6460 2023-07-14T17:44:07.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.161+0000 7ff77d971b80 10 client.0 _ll_register_callbacks cb 0x55819a493710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:07.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff76affd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 0x55819a5ae8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff76affd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 0x55819a5ae8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42850/0 (socket says 172.21.15.88:42850) 2023-07-14T17:44:07.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff76affd700 1 -- 172.21.15.88:0/3886913831 learned_addr learned my addr 172.21.15.88:0/3886913831 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:07.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff76a7fc700 1 --2- 172.21.15.88:0/3886913831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a5860 0x55819a5aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff76b7fe700 1 --2- 172.21.15.88:0/3886913831 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a6460 0x55819a5b5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff76affd700 1 -- 172.21.15.88:0/3886913831 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a6460 msgr2=0x55819a5b5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:07.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff76affd700 1 --2- 172.21.15.88:0/3886913831 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a6460 0x55819a5b5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff76affd700 1 -- 172.21.15.88:0/3886913831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a5860 msgr2=0x55819a5aee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:07.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff76affd700 1 --2- 172.21.15.88:0/3886913831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a5860 0x55819a5aee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff76affd700 1 -- 172.21.15.88:0/3886913831 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff75c009940 con 0x55819a5a4f10 2023-07-14T17:44:07.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff76affd700 1 --2- 172.21.15.88:0/3886913831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 0x55819a5ae8f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff76000ce00 tx=0x7ff760004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f13f2c0e0fd1afa4 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:07.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff74eff5700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:07.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff74eff5700 1 -- 172.21.15.88:0/3886913831 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76000cee0 con 0x55819a5a4f10 2023-07-14T17:44:07.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff74eff5700 1 -- 172.21.15.88:0/3886913831 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff76001f3f0 con 0x55819a5a4f10 2023-07-14T17:44:07.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff77d971b80 1 -- 172.21.15.88:0/3886913831 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55819a44f4a0 con 0x55819a5a4f10 2023-07-14T17:44:07.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff74eff5700 1 -- 172.21.15.88:0/3886913831 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff760013e50 con 0x55819a5a4f10 2023-07-14T17:44:07.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff77d971b80 10 client.15285 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:07.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff77d971b80 1 -- 172.21.15.88:0/3886913831 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55819a30cc10 con 0x55819a5a4f10 2023-07-14T17:44:07.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.162+0000 7ff74eff5700 1 -- 172.21.15.88:0/3886913831 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55819a44f4a0 con 0x55819a5a4f10 2023-07-14T17:44:07.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.163+0000 7ff74eff5700 1 -- 172.21.15.88:0/3886913831 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff76001e070 con 0x55819a5a4f10 2023-07-14T17:44:07.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.163+0000 7ff77d971b80 10 client.15285 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:07.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.163+0000 7ff77d971b80 10 client.15285 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T17:44:07.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.163+0000 7ff77d971b80 20 client.15285 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:07.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.163+0000 7ff77d971b80 1 --2- 172.21.15.88:0/3886913831 >> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] conn(0x55819a5b9be0 0x55819a5b8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.163+0000 7ff77d971b80 4 client.15285 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T17:44:07.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.163+0000 7ff77d971b80 1 -- 172.21.15.88:0/3886913831 --> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55819a573d50 con 0x55819a5b9be0 2023-07-14T17:44:07.156 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.163+0000 7ff76a7fc700 1 --2- 172.21.15.88:0/3886913831 >> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] conn(0x55819a5b9be0 0x55819a5b8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.164+0000 7ff74eff5700 10 client.15285 ms_handle_connect on v2:172.21.15.107:6836/1152608615 2023-07-14T17:44:07.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.164+0000 7ff76a7fc700 1 --2- 172.21.15.88:0/3886913831 >> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] conn(0x55819a5b9be0 0x55819a5b8f90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7f73b1ed0b7fc23 server_cookie=88f0ee3ae0760 in_seq=0 out_seq=0 2023-07-14T17:44:07.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.164+0000 7ff74eff5700 1 -- 172.21.15.88:0/3886913831 <== mds.1 v2:172.21.15.107:6836/1152608615 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55819a573d50 con 0x55819a5b9be0 2023-07-14T17:44:07.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.164+0000 7ff74eff5700 10 client.15285 handle_command_reply: tid=0 2023-07-14T17:44:07.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.164+0000 7ff77d971b80 1 client.15285 shutdown 2023-07-14T17:44:07.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.164+0000 7ff77d971b80 1 -- 172.21.15.88:0/3886913831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 msgr2=0x55819a5ae8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.164+0000 7ff77d971b80 1 --2- 172.21.15.88:0/3886913831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 0x55819a5ae8f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff76000ce00 tx=0x7ff760004860 comp rx=0 tx=0).stop 2023-07-14T17:44:07.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.164+0000 7ff77d971b80 1 -- 172.21.15.88:0/3886913831 shutdown_connections 2023-07-14T17:44:07.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.164+0000 7ff77d971b80 1 --2- 172.21.15.88:0/3886913831 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a6460 0x55819a5b5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.164+0000 7ff77d971b80 1 --2- 172.21.15.88:0/3886913831 >> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] conn(0x55819a5b9be0 0x55819a5b8f90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.165+0000 7ff77d971b80 1 --2- 172.21.15.88:0/3886913831 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 0x55819a5ae8f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.165+0000 7ff74eff5700 0 client.15285 ms_handle_reset on v2:172.21.15.107:6836/1152608615 2023-07-14T17:44:07.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.165+0000 7ff77d971b80 1 --2- 172.21.15.88:0/3886913831 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a5860 0x55819a5aee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.165+0000 7ff77d971b80 1 -- 172.21.15.88:0/3886913831 >> 172.21.15.88:0/3886913831 conn(0x55819a59a630 msgr2=0x55819a59e600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:07.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.165+0000 7ff77d971b80 1 -- 172.21.15.88:0/3886913831 shutdown_connections 2023-07-14T17:44:07.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.165+0000 7ff77d971b80 1 -- 172.21.15.88:0/3886913831 wait complete. 2023-07-14T17:44:07.160 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.165+0000 7ff77d971b80 20 client.15285 trim_cache size 0 max 16384 2023-07-14T17:44:07.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.182+0000 7ff77d971b80 1 Processor -- start 2023-07-14T17:44:07.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.182+0000 7ff77d971b80 1 -- start start 2023-07-14T17:44:07.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.182+0000 7ff77d971b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 0x55819a5b66a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.182+0000 7ff77d971b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a5860 0x55819a5b8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.182+0000 7ff77d971b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a6460 0x55819a5b94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.182+0000 7ff77d971b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55819a5ae4c0 con 0x55819a5a4f10 2023-07-14T17:44:07.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.182+0000 7ff77d971b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55819a44f4a0 con 0x55819a5a6460 2023-07-14T17:44:07.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.182+0000 7ff77d971b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55819a4c4790 con 0x55819a5a5860 2023-07-14T17:44:07.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.182+0000 7ff76affd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 0x55819a5b66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.182+0000 7ff76affd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 0x55819a5b66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42858/0 (socket says 172.21.15.88:42858) 2023-07-14T17:44:07.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.182+0000 7ff76affd700 1 -- 172.21.15.88:0/594306218 learned_addr learned my addr 172.21.15.88:0/594306218 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:07.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.182+0000 7ff76b7fe700 1 --2- 172.21.15.88:0/594306218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a6460 0x55819a5b94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.182+0000 7ff76affd700 1 -- 172.21.15.88:0/594306218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a5860 msgr2=0x55819a5b8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:07.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.182+0000 7ff76affd700 1 --2- 172.21.15.88:0/594306218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a5860 0x55819a5b8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.182+0000 7ff76affd700 1 -- 172.21.15.88:0/594306218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a6460 msgr2=0x55819a5b94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:07.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.182+0000 7ff76affd700 1 --2- 172.21.15.88:0/594306218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a6460 0x55819a5b94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.182+0000 7ff76affd700 1 -- 172.21.15.88:0/594306218 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55819a59ff00 con 0x55819a5a4f10 2023-07-14T17:44:07.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.182+0000 7ff76b7fe700 1 --2- 172.21.15.88:0/594306218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a6460 0x55819a5b94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:07.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.183+0000 7ff76affd700 1 --2- 172.21.15.88:0/594306218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 0x55819a5b66a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff760000c00 tx=0x7ff76000c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=242ac72e584ac5b1 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:07.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.183+0000 7ff751ffb700 1 -- 172.21.15.88:0/594306218 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff760007dc0 con 0x55819a5a4f10 2023-07-14T17:44:07.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.183+0000 7ff751ffb700 1 -- 172.21.15.88:0/594306218 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff760013e50 con 0x55819a5a4f10 2023-07-14T17:44:07.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.183+0000 7ff751ffb700 1 -- 172.21.15.88:0/594306218 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76001fb70 con 0x55819a5a4f10 2023-07-14T17:44:07.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.183+0000 7ff77d971b80 1 -- 172.21.15.88:0/594306218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 msgr2=0x55819a5b66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.183+0000 7ff77d971b80 1 --2- 172.21.15.88:0/594306218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 0x55819a5b66a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff760000c00 tx=0x7ff76000c430 comp rx=0 tx=0).stop 2023-07-14T17:44:07.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.183+0000 7ff77d971b80 1 -- 172.21.15.88:0/594306218 shutdown_connections 2023-07-14T17:44:07.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.183+0000 7ff77d971b80 1 --2- 172.21.15.88:0/594306218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a6460 0x55819a5b94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.183+0000 7ff77d971b80 1 --2- 172.21.15.88:0/594306218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 0x55819a5b66a0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.183+0000 7ff77d971b80 1 --2- 172.21.15.88:0/594306218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a5860 0x55819a5b8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.183+0000 7ff77d971b80 1 -- 172.21.15.88:0/594306218 >> 172.21.15.88:0/594306218 conn(0x55819a59a630 msgr2=0x55819a5ac810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:07.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.183+0000 7ff77d971b80 1 -- 172.21.15.88:0/594306218 shutdown_connections 2023-07-14T17:44:07.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.183+0000 7ff77d971b80 1 -- 172.21.15.88:0/594306218 wait complete. 2023-07-14T17:44:07.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff77d971b80 1 -- start start 2023-07-14T17:44:07.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff77d971b80 1 Processor -- start 2023-07-14T17:44:07.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff77d971b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 0x55819a5b18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff77d971b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a5860 0x55819a60f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff77d971b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a6460 0x55819a611ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff77d971b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55819a612610 con 0x55819a5a4f10 2023-07-14T17:44:07.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff77d971b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55819a612780 con 0x55819a5a6460 2023-07-14T17:44:07.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff77d971b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55819a612940 con 0x55819a5a5860 2023-07-14T17:44:07.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff77d971b80 10 client.0 _ll_register_callbacks cb 0x55819a493710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:07.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff76affd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 0x55819a5b18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff76affd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 0x55819a5b18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42864/0 (socket says 172.21.15.88:42864) 2023-07-14T17:44:07.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff76affd700 1 -- 172.21.15.88:0/2179181182 learned_addr learned my addr 172.21.15.88:0/2179181182 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:07.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff76a7fc700 1 --2- 172.21.15.88:0/2179181182 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a5860 0x55819a60f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff76affd700 1 -- 172.21.15.88:0/2179181182 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a5860 msgr2=0x55819a60f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:07.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff76affd700 1 --2- 172.21.15.88:0/2179181182 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a5860 0x55819a60f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff76affd700 1 -- 172.21.15.88:0/2179181182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a6460 msgr2=0x55819a611ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:07.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff76affd700 1 --2- 172.21.15.88:0/2179181182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a6460 0x55819a611ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff76affd700 1 -- 172.21.15.88:0/2179181182 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55819a30cc10 con 0x55819a5a4f10 2023-07-14T17:44:07.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff76a7fc700 1 --2- 172.21.15.88:0/2179181182 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a5860 0x55819a60f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:07.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff76affd700 1 --2- 172.21.15.88:0/2179181182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 0x55819a5b18b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff76000e010 tx=0x7ff760011680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e4c2561192d3e6 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:07.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff74eff5700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:07.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff74eff5700 1 -- 172.21.15.88:0/2179181182 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7600119f0 con 0x55819a5a4f10 2023-07-14T17:44:07.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff74eff5700 1 -- 172.21.15.88:0/2179181182 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff76001f8f0 con 0x55819a5a4f10 2023-07-14T17:44:07.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.184+0000 7ff77d971b80 1 -- 172.21.15.88:0/2179181182 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff75c00ecd0 con 0x55819a5a4f10 2023-07-14T17:44:07.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.185+0000 7ff74eff5700 1 -- 172.21.15.88:0/2179181182 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff760013a90 con 0x55819a5a4f10 2023-07-14T17:44:07.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.185+0000 7ff77d971b80 10 client.15291 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:07.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.185+0000 7ff77d971b80 1 -- 172.21.15.88:0/2179181182 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55819a34a200 con 0x55819a5a4f10 2023-07-14T17:44:07.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.185+0000 7ff74eff5700 1 -- 172.21.15.88:0/2179181182 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff75c00ecd0 con 0x55819a5a4f10 2023-07-14T17:44:07.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.185+0000 7ff74eff5700 1 -- 172.21.15.88:0/2179181182 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff76001e070 con 0x55819a5a4f10 2023-07-14T17:44:07.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.185+0000 7ff77d971b80 10 client.15291 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:07.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.185+0000 7ff77d971b80 10 client.15291 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T17:44:07.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.185+0000 7ff77d971b80 20 client.15291 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:07.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.185+0000 7ff77d971b80 1 --2- 172.21.15.88:0/2179181182 >> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] conn(0x55819a6172e0 0x55819a613bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.176 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.185+0000 7ff77d971b80 4 client.15291 mds_command: new command op to 24473 tid=0[{"prefix": "status"}] 2023-07-14T17:44:07.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.185+0000 7ff77d971b80 1 -- 172.21.15.88:0/2179181182 --> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55819a4eee60 con 0x55819a6172e0 2023-07-14T17:44:07.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.185+0000 7ff76a7fc700 1 --2- 172.21.15.88:0/2179181182 >> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] conn(0x55819a6172e0 0x55819a613bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.186+0000 7ff76a7fc700 1 --2- 172.21.15.88:0/2179181182 >> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] conn(0x55819a6172e0 0x55819a613bd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8577cbd261efb526 server_cookie=6bde6db88072d7b9 in_seq=0 out_seq=0 2023-07-14T17:44:07.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.186+0000 7ff74eff5700 10 client.15291 ms_handle_connect on v2:172.21.15.107:6836/1152608615 2023-07-14T17:44:07.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.186+0000 7ff74eff5700 1 -- 172.21.15.88:0/2179181182 <== mds.1 v2:172.21.15.107:6836/1152608615 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55819a4eee60 con 0x55819a6172e0 2023-07-14T17:44:07.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.186+0000 7ff74eff5700 10 client.15291 handle_command_reply: tid=0 2023-07-14T17:44:07.177 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.186+0000 7ff77d971b80 1 client.15291 shutdown 2023-07-14T17:44:07.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.186+0000 7ff77d971b80 1 -- 172.21.15.88:0/2179181182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 msgr2=0x55819a5b18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.186+0000 7ff77d971b80 1 --2- 172.21.15.88:0/2179181182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 0x55819a5b18b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff76000e010 tx=0x7ff760011680 comp rx=0 tx=0).stop 2023-07-14T17:44:07.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.186+0000 7ff77d971b80 1 -- 172.21.15.88:0/2179181182 shutdown_connections 2023-07-14T17:44:07.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.186+0000 7ff77d971b80 1 --2- 172.21.15.88:0/2179181182 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55819a5a6460 0x55819a611ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.186+0000 7ff77d971b80 1 --2- 172.21.15.88:0/2179181182 >> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] conn(0x55819a6172e0 0x55819a613bd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.186+0000 7ff77d971b80 1 --2- 172.21.15.88:0/2179181182 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55819a5a4f10 0x55819a5b18b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.186+0000 7ff77d971b80 1 --2- 172.21.15.88:0/2179181182 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55819a5a5860 0x55819a60f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.186+0000 7ff77d971b80 1 -- 172.21.15.88:0/2179181182 >> 172.21.15.88:0/2179181182 conn(0x55819a59a630 msgr2=0x55819a5b66a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:07.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.186+0000 7ff74eff5700 0 client.15291 ms_handle_reset on v2:172.21.15.107:6836/1152608615 2023-07-14T17:44:07.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.186+0000 7ff77d971b80 1 -- 172.21.15.88:0/2179181182 shutdown_connections 2023-07-14T17:44:07.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.186+0000 7ff77d971b80 1 -- 172.21.15.88:0/2179181182 wait complete. 2023-07-14T17:44:07.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.186+0000 7ff77d971b80 20 client.15291 trim_cache size 0 max 16384 2023-07-14T17:44:07.179 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-07-14T17:44:07.180 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "b462a556-226c-11ee-9b33-001a4aab830c", 2023-07-14T17:44:07.180 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 1, 2023-07-14T17:44:07.180 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 24473, 2023-07-14T17:44:07.180 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2023-07-14T17:44:07.180 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2023-07-14T17:44:07.180 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2023-07-14T17:44:07.180 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 66.453347684999997, 2023-07-14T17:44:07.181 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 27, 2023-07-14T17:44:07.181 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 88, 2023-07-14T17:44:07.181 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 86, 2023-07-14T17:44:07.181 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 98.452083927000004 2023-07-14T17:44:07.181 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-07-14T17:44:07.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.187+0000 7ff7717d9700 1 -- 172.21.15.88:0/588275649 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff7540a11e0 msgr2=0x7ff7540a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.187+0000 7ff7717d9700 1 --2- 172.21.15.88:0/588275649 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff7540a11e0 0x7ff7540a3690 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff76c0ffb60 tx=0x7ff760008040 comp rx=0 tx=0).stop 2023-07-14T17:44:07.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.187+0000 7ff7717d9700 1 -- 172.21.15.88:0/588275649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff76c100690 msgr2=0x7ff76c19cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.187+0000 7ff7717d9700 1 --2- 172.21.15.88:0/588275649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff76c100690 0x7ff76c19cfb0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff76400b830 tx=0x7ff764004d90 comp rx=0 tx=0).stop 2023-07-14T17:44:07.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.187+0000 7ff7717d9700 1 -- 172.21.15.88:0/588275649 shutdown_connections 2023-07-14T17:44:07.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.187+0000 7ff7717d9700 1 --2- 172.21.15.88:0/588275649 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff7540a11e0 0x7ff7540a3690 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.187+0000 7ff7717d9700 1 --2- 172.21.15.88:0/588275649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff76c100690 0x7ff76c19cfb0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.187+0000 7ff7717d9700 1 --2- 172.21.15.88:0/588275649 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff76c0feb10 0x7ff76c195a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.187+0000 7ff7717d9700 1 --2- 172.21.15.88:0/588275649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff76c0ffd00 0x7ff76c195f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.187+0000 7ff7717d9700 1 -- 172.21.15.88:0/588275649 >> 172.21.15.88:0/588275649 conn(0x7ff76c0fa130 msgr2=0x7ff76c0fc320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:07.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.187+0000 7ff7717d9700 1 -- 172.21.15.88:0/588275649 shutdown_connections 2023-07-14T17:44:07.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.187+0000 7ff7717d9700 1 -- 172.21.15.88:0/588275649 wait complete. 2023-07-14T17:44:07.184 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:06 smithi088 ceph-mon[127465]: pgmap v299: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2023-07-14T17:44:07.184 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:06 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/685600365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:07.184 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:06 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2242484567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:07.188 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-14T17:44:07.223 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:06 smithi107 ceph-mon[128999]: pgmap v299: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2023-07-14T17:44:07.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:06 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/685600365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:07.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:06 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2242484567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:07.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.329+0000 7ff2a94e8700 1 -- 172.21.15.88:0/1275196642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a40fdf20 msgr2=0x7ff2a40fe330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.329+0000 7ff2a94e8700 1 --2- 172.21.15.88:0/1275196642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a40fdf20 0x7ff2a40fe330 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7ff298009990 tx=0x7ff298009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.329+0000 7ff2a94e8700 1 -- 172.21.15.88:0/1275196642 shutdown_connections 2023-07-14T17:44:07.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.329+0000 7ff2a94e8700 1 --2- 172.21.15.88:0/1275196642 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff2a4101920 0x7ff2a41081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.329+0000 7ff2a94e8700 1 --2- 172.21.15.88:0/1275196642 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a40fdf20 0x7ff2a40fe330 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.329+0000 7ff2a94e8700 1 --2- 172.21.15.88:0/1275196642 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2a4100f90 0x7ff2a41013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.329+0000 7ff2a94e8700 1 -- 172.21.15.88:0/1275196642 >> 172.21.15.88:0/1275196642 conn(0x7ff2a40f98e0 msgr2=0x7ff2a40fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:07.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.329+0000 7ff2a94e8700 1 -- 172.21.15.88:0/1275196642 shutdown_connections 2023-07-14T17:44:07.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.329+0000 7ff2a94e8700 1 -- 172.21.15.88:0/1275196642 wait complete. 2023-07-14T17:44:07.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.329+0000 7ff2a94e8700 1 Processor -- start 2023-07-14T17:44:07.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a94e8700 1 -- start start 2023-07-14T17:44:07.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a94e8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a40fdf20 0x7ff2a4195870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a94e8700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff2a4100f90 0x7ff2a4195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a94e8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2a4101920 0x7ff2a419ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a94e8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a419d3f0 con 0x7ff2a40fdf20 2023-07-14T17:44:07.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a94e8700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a419d530 con 0x7ff2a4101920 2023-07-14T17:44:07.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a94e8700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a419d6c0 con 0x7ff2a4100f90 2023-07-14T17:44:07.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a2ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a40fdf20 0x7ff2a4195870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a2ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a40fdf20 0x7ff2a4195870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42878/0 (socket says 172.21.15.88:42878) 2023-07-14T17:44:07.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a2ffd700 1 -- 172.21.15.88:0/2210241274 learned_addr learned my addr 172.21.15.88:0/2210241274 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:07.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a27fc700 1 --2- 172.21.15.88:0/2210241274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff2a4100f90 0x7ff2a4195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a37fe700 1 --2- 172.21.15.88:0/2210241274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2a4101920 0x7ff2a419ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a2ffd700 1 -- 172.21.15.88:0/2210241274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff2a4100f90 msgr2=0x7ff2a4195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a2ffd700 1 --2- 172.21.15.88:0/2210241274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff2a4100f90 0x7ff2a4195db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a2ffd700 1 -- 172.21.15.88:0/2210241274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2a4101920 msgr2=0x7ff2a419ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a2ffd700 1 --2- 172.21.15.88:0/2210241274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2a4101920 0x7ff2a419ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a2ffd700 1 -- 172.21.15.88:0/2210241274 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2980095f0 con 0x7ff2a40fdf20 2023-07-14T17:44:07.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a37fe700 1 --2- 172.21.15.88:0/2210241274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2a4101920 0x7ff2a419ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:07.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff2a2ffd700 1 --2- 172.21.15.88:0/2210241274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a40fdf20 0x7ff2a4195870 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff29800c010 tx=0x7ff298004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:07.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.330+0000 7ff28bfff700 1 -- 172.21.15.88:0/2210241274 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29801d070 con 0x7ff2a40fdf20 2023-07-14T17:44:07.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.331+0000 7ff2a94e8700 1 -- 172.21.15.88:0/2210241274 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a419d940 con 0x7ff2a40fdf20 2023-07-14T17:44:07.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.331+0000 7ff28bfff700 1 -- 172.21.15.88:0/2210241274 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff2980054c0 con 0x7ff2a40fdf20 2023-07-14T17:44:07.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.331+0000 7ff28bfff700 1 -- 172.21.15.88:0/2210241274 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff298003ab0 con 0x7ff2a40fdf20 2023-07-14T17:44:07.317 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.331+0000 7ff2a94e8700 1 -- 172.21.15.88:0/2210241274 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a419ddc0 con 0x7ff2a40fdf20 2023-07-14T17:44:07.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.333+0000 7ff28bfff700 1 -- 172.21.15.88:0/2210241274 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff2980047a0 con 0x7ff2a40fdf20 2023-07-14T17:44:07.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.334+0000 7ff28bfff700 1 --2- 172.21.15.88:0/2210241274 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff28c0a11e0 0x7ff28c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.334+0000 7ff2a27fc700 1 --2- 172.21.15.88:0/2210241274 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff28c0a11e0 0x7ff28c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.334+0000 7ff28bfff700 1 -- 172.21.15.88:0/2210241274 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff2980cdd50 con 0x7ff2a40fdf20 2023-07-14T17:44:07.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.334+0000 7ff2b5680b80 1 Processor -- start 2023-07-14T17:44:07.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.334+0000 7ff2b5680b80 1 -- start start 2023-07-14T17:44:07.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.334+0000 7ff2a27fc700 1 --2- 172.21.15.88:0/2210241274 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff28c0a11e0 0x7ff28c0a3690 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff294005950 tx=0x7ff29400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:07.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.334+0000 7ff2b5680b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779b9f10 0x562d779ba320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.334+0000 7ff2b5680b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 0x562d779bae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.334+0000 7ff2b5680b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779bb460 0x562d779bb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.334+0000 7ff2b5680b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d778bec60 con 0x562d779ba860 2023-07-14T17:44:07.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.334+0000 7ff2b5680b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d779119f0 con 0x562d779bb460 2023-07-14T17:44:07.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.334+0000 7ff2b5680b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d7790ae80 con 0x562d779b9f10 2023-07-14T17:44:07.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.334+0000 7ff2a27fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 0x562d779bae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.334+0000 7ff2a27fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 0x562d779bae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42888/0 (socket says 172.21.15.88:42888) 2023-07-14T17:44:07.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.334+0000 7ff2a27fc700 1 -- 172.21.15.88:0/3927066308 learned_addr learned my addr 172.21.15.88:0/3927066308 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:07.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.334+0000 7ff2a37fe700 1 --2- 172.21.15.88:0/3927066308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779bb460 0x562d779bb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.335+0000 7ff2a2ffd700 1 --2- 172.21.15.88:0/3927066308 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779b9f10 0x562d779ba320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.335+0000 7ff2a27fc700 1 -- 172.21.15.88:0/3927066308 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779b9f10 msgr2=0x562d779ba320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:07.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.335+0000 7ff2a27fc700 1 --2- 172.21.15.88:0/3927066308 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779b9f10 0x562d779ba320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.335+0000 7ff2a27fc700 1 -- 172.21.15.88:0/3927066308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779bb460 msgr2=0x562d779bb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:07.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.335+0000 7ff2a27fc700 1 --2- 172.21.15.88:0/3927066308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779bb460 0x562d779bb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.335+0000 7ff2a27fc700 1 -- 172.21.15.88:0/3927066308 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562d77738d90 con 0x562d779ba860 2023-07-14T17:44:07.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.335+0000 7ff2a27fc700 1 --2- 172.21.15.88:0/3927066308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 0x562d779bae80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff29400fde0 tx=0x7ff294012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae35b035d4da4d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:07.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.335+0000 7ff2897fa700 1 -- 172.21.15.88:0/3927066308 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2940108c0 con 0x562d779ba860 2023-07-14T17:44:07.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.335+0000 7ff2897fa700 1 -- 172.21.15.88:0/3927066308 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff294020070 con 0x562d779ba860 2023-07-14T17:44:07.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.335+0000 7ff2897fa700 1 -- 172.21.15.88:0/3927066308 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294025d50 con 0x562d779ba860 2023-07-14T17:44:07.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.336+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3927066308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 msgr2=0x562d779bae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.336+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/3927066308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 0x562d779bae80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff29400fde0 tx=0x7ff294012000 comp rx=0 tx=0).stop 2023-07-14T17:44:07.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.336+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3927066308 shutdown_connections 2023-07-14T17:44:07.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.336+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/3927066308 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779bb460 0x562d779bb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.336+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/3927066308 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779b9f10 0x562d779ba320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.336+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/3927066308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 0x562d779bae80 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.336+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3927066308 >> 172.21.15.88:0/3927066308 conn(0x562d779af630 msgr2=0x562d779b4980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:07.327 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.336+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3927066308 shutdown_connections 2023-07-14T17:44:07.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.336+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3927066308 wait complete. 2023-07-14T17:44:07.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.336+0000 7ff2b5680b80 1 -- start start 2023-07-14T17:44:07.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.336+0000 7ff2b5680b80 1 Processor -- start 2023-07-14T17:44:07.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.336+0000 7ff2b5680b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779b9f10 0x562d779c39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.336+0000 7ff2b5680b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779ba860 0x562d779c3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.328 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.336+0000 7ff2b5680b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779bb460 0x562d779caf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.336+0000 7ff2b5680b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d778aaa70 con 0x562d779b9f10 2023-07-14T17:44:07.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.336+0000 7ff2b5680b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d778aa770 con 0x562d779bb460 2023-07-14T17:44:07.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.336+0000 7ff2b5680b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d7778e8d0 con 0x562d779ba860 2023-07-14T17:44:07.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.336+0000 7ff2b5680b80 10 client.0 _ll_register_callbacks cb 0x562d778a8710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:07.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff2a2ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779b9f10 0x562d779c39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff2a2ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779b9f10 0x562d779c39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42892/0 (socket says 172.21.15.88:42892) 2023-07-14T17:44:07.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff2a2ffd700 1 -- 172.21.15.88:0/3968366041 learned_addr learned my addr 172.21.15.88:0/3968366041 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:07.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff2a37fe700 1 --2- 172.21.15.88:0/3968366041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779bb460 0x562d779caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff2a27fc700 1 --2- 172.21.15.88:0/3968366041 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779ba860 0x562d779c3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff2a2ffd700 1 -- 172.21.15.88:0/3968366041 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779ba860 msgr2=0x562d779c3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:07.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff2a2ffd700 1 --2- 172.21.15.88:0/3968366041 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779ba860 0x562d779c3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff2a2ffd700 1 -- 172.21.15.88:0/3968366041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779bb460 msgr2=0x562d779caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:07.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff2a2ffd700 1 --2- 172.21.15.88:0/3968366041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779bb460 0x562d779caf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff2a2ffd700 1 -- 172.21.15.88:0/3968366041 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29400fa90 con 0x562d779b9f10 2023-07-14T17:44:07.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff2a37fe700 1 --2- 172.21.15.88:0/3968366041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779bb460 0x562d779caf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:07.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff2a2ffd700 1 --2- 172.21.15.88:0/3968366041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779b9f10 0x562d779c39a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff2980977f0 tx=0x7ff29809b2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3196a75aed30a778 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:07.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff286ff5700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:07.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff286ff5700 1 -- 172.21.15.88:0/3968366041 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2980a3140 con 0x562d779b9f10 2023-07-14T17:44:07.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3968366041 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562d778644a0 con 0x562d779b9f10 2023-07-14T17:44:07.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff286ff5700 1 -- 172.21.15.88:0/3968366041 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff2980b34a0 con 0x562d779b9f10 2023-07-14T17:44:07.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff286ff5700 1 -- 172.21.15.88:0/3968366041 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2980a2cb0 con 0x562d779b9f10 2023-07-14T17:44:07.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff2b5680b80 10 client.15303 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:07.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3968366041 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x562d77721c10 con 0x562d779b9f10 2023-07-14T17:44:07.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.337+0000 7ff286ff5700 1 -- 172.21.15.88:0/3968366041 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x562d778644a0 con 0x562d779b9f10 2023-07-14T17:44:07.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.338+0000 7ff286ff5700 1 -- 172.21.15.88:0/3968366041 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff298099ce0 con 0x562d779b9f10 2023-07-14T17:44:07.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.338+0000 7ff2b5680b80 10 client.15303 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:07.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.338+0000 7ff2b5680b80 10 client.15303 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T17:44:07.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.338+0000 7ff2b5680b80 20 client.15303 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:07.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.338+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/3968366041 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x562d779ce880 0x562d779cde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.332 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.338+0000 7ff2b5680b80 4 client.15303 mds_command: new command op to 24506 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T17:44:07.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.338+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3968366041 --> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562d77988d50 con 0x562d779ce880 2023-07-14T17:44:07.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.338+0000 7ff2a27fc700 1 --2- 172.21.15.88:0/3968366041 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x562d779ce880 0x562d779cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.338+0000 7ff286ff5700 10 client.15303 ms_handle_connect on v2:172.21.15.107:6838/31646548 2023-07-14T17:44:07.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.338+0000 7ff2a27fc700 1 --2- 172.21.15.88:0/3968366041 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x562d779ce880 0x562d779cde50 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7eb188e678615054 server_cookie=d6abb1716971cbff in_seq=0 out_seq=0 2023-07-14T17:44:07.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.339+0000 7ff286ff5700 1 -- 172.21.15.88:0/3968366041 <== mds.0 v2:172.21.15.107:6838/31646548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x562d77988d50 con 0x562d779ce880 2023-07-14T17:44:07.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.339+0000 7ff286ff5700 10 client.15303 handle_command_reply: tid=0 2023-07-14T17:44:07.333 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.339+0000 7ff2b5680b80 1 client.15303 shutdown 2023-07-14T17:44:07.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.339+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3968366041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779b9f10 msgr2=0x562d779c39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.339+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/3968366041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779b9f10 0x562d779c39a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff2980977f0 tx=0x7ff29809b2c0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.339+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3968366041 shutdown_connections 2023-07-14T17:44:07.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.339+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/3968366041 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x562d779ce880 0x562d779cde50 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.339+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/3968366041 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779bb460 0x562d779caf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.339+0000 7ff286ff5700 0 client.15303 ms_handle_reset on v2:172.21.15.107:6838/31646548 2023-07-14T17:44:07.334 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.339+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/3968366041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779b9f10 0x562d779c39a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.339+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/3968366041 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779ba860 0x562d779c3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.339+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3968366041 >> 172.21.15.88:0/3968366041 conn(0x562d779af630 msgr2=0x562d779b3600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:07.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.339+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3968366041 shutdown_connections 2023-07-14T17:44:07.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.339+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3968366041 wait complete. 2023-07-14T17:44:07.335 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.339+0000 7ff2b5680b80 20 client.15303 trim_cache size 0 max 16384 2023-07-14T17:44:07.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.357+0000 7ff2b5680b80 1 Processor -- start 2023-07-14T17:44:07.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.357+0000 7ff2b5680b80 1 -- start start 2023-07-14T17:44:07.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.357+0000 7ff2b5680b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779b9f10 0x562d779b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.340 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.357+0000 7ff2b5680b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 0x562d779caee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.357+0000 7ff2b5680b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779bb460 0x562d779cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.357+0000 7ff2b5680b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d778644a0 con 0x562d779ba860 2023-07-14T17:44:07.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.357+0000 7ff2b5680b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d778b4470 con 0x562d779b9f10 2023-07-14T17:44:07.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.357+0000 7ff2b5680b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d7779ade0 con 0x562d779bb460 2023-07-14T17:44:07.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.357+0000 7ff2a27fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 0x562d779caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.357+0000 7ff2a27fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 0x562d779caee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42900/0 (socket says 172.21.15.88:42900) 2023-07-14T17:44:07.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.357+0000 7ff2a27fc700 1 -- 172.21.15.88:0/2046102794 learned_addr learned my addr 172.21.15.88:0/2046102794 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:07.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.357+0000 7ff2a27fc700 1 -- 172.21.15.88:0/2046102794 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779bb460 msgr2=0x562d779cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:07.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.357+0000 7ff2a27fc700 1 --2- 172.21.15.88:0/2046102794 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779bb460 0x562d779cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.357+0000 7ff2a27fc700 1 -- 172.21.15.88:0/2046102794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779b9f10 msgr2=0x562d779b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:07.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.357+0000 7ff2a27fc700 1 --2- 172.21.15.88:0/2046102794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779b9f10 0x562d779b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.357+0000 7ff2a27fc700 1 -- 172.21.15.88:0/2046102794 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562d779b4f00 con 0x562d779ba860 2023-07-14T17:44:07.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.357+0000 7ff2a27fc700 1 --2- 172.21.15.88:0/2046102794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 0x562d779caee0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff29400fa60 tx=0x7ff29400a370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b67cfed4368b2f4e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:07.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.358+0000 7ff289ffb700 1 -- 172.21.15.88:0/2046102794 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29402b040 con 0x562d779ba860 2023-07-14T17:44:07.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.358+0000 7ff289ffb700 1 -- 172.21.15.88:0/2046102794 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff294014070 con 0x562d779ba860 2023-07-14T17:44:07.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.358+0000 7ff289ffb700 1 -- 172.21.15.88:0/2046102794 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29402f670 con 0x562d779ba860 2023-07-14T17:44:07.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.358+0000 7ff2b5680b80 1 -- 172.21.15.88:0/2046102794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 msgr2=0x562d779caee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.358+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/2046102794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 0x562d779caee0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff29400fa60 tx=0x7ff29400a370 comp rx=0 tx=0).stop 2023-07-14T17:44:07.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.358+0000 7ff2b5680b80 1 -- 172.21.15.88:0/2046102794 shutdown_connections 2023-07-14T17:44:07.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.358+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/2046102794 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779bb460 0x562d779cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.358+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/2046102794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779b9f10 0x562d779b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.358+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/2046102794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 0x562d779caee0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.358+0000 7ff2b5680b80 1 -- 172.21.15.88:0/2046102794 >> 172.21.15.88:0/2046102794 conn(0x562d779af630 msgr2=0x562d779c2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:07.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.358+0000 7ff2b5680b80 1 -- 172.21.15.88:0/2046102794 shutdown_connections 2023-07-14T17:44:07.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.358+0000 7ff2b5680b80 1 -- 172.21.15.88:0/2046102794 wait complete. 2023-07-14T17:44:07.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.358+0000 7ff2b5680b80 1 -- start start 2023-07-14T17:44:07.344 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2b5680b80 1 Processor -- start 2023-07-14T17:44:07.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2b5680b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779b9f10 0x562d779c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.347 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2b5680b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 0x562d779c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2b5680b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779bb460 0x562d779c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2b5680b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d779ca080 con 0x562d779ba860 2023-07-14T17:44:07.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2b5680b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d779ca1f0 con 0x562d779b9f10 2023-07-14T17:44:07.348 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2b5680b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d779ca3b0 con 0x562d779bb460 2023-07-14T17:44:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2b5680b80 10 client.0 _ll_register_callbacks cb 0x7ff29809a6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2a27fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 0x562d779c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2a27fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 0x562d779c6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42908/0 (socket says 172.21.15.88:42908) 2023-07-14T17:44:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2a27fc700 1 -- 172.21.15.88:0/3054578727 learned_addr learned my addr 172.21.15.88:0/3054578727 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2a37fe700 1 --2- 172.21.15.88:0/3054578727 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779bb460 0x562d779c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2a2ffd700 1 --2- 172.21.15.88:0/3054578727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779b9f10 0x562d779c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.349 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2a27fc700 1 -- 172.21.15.88:0/3054578727 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779bb460 msgr2=0x562d779c9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2a27fc700 1 --2- 172.21.15.88:0/3054578727 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779bb460 0x562d779c9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2a27fc700 1 -- 172.21.15.88:0/3054578727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779b9f10 msgr2=0x562d779c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2a27fc700 1 --2- 172.21.15.88:0/3054578727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779b9f10 0x562d779c67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2a27fc700 1 -- 172.21.15.88:0/3054578727 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562d77738d90 con 0x562d779ba860 2023-07-14T17:44:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2a2ffd700 1 --2- 172.21.15.88:0/3054578727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779b9f10 0x562d779c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2a27fc700 1 --2- 172.21.15.88:0/3054578727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 0x562d779c6cf0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff294031040 tx=0x7ff29400ff40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b393a302787db5 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:07.350 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff286ff5700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:07.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff286ff5700 1 -- 172.21.15.88:0/3054578727 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29402b040 con 0x562d779ba860 2023-07-14T17:44:07.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff286ff5700 1 -- 172.21.15.88:0/3054578727 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff29402fc50 con 0x562d779ba860 2023-07-14T17:44:07.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.359+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3054578727 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff294020070 con 0x562d779ba860 2023-07-14T17:44:07.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.360+0000 7ff286ff5700 1 -- 172.21.15.88:0/3054578727 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff29402fe10 con 0x562d779ba860 2023-07-14T17:44:07.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.360+0000 7ff2b5680b80 10 client.15309 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:07.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.360+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3054578727 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x562d7775f200 con 0x562d779ba860 2023-07-14T17:44:07.351 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.360+0000 7ff286ff5700 1 -- 172.21.15.88:0/3054578727 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff294020070 con 0x562d779ba860 2023-07-14T17:44:07.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.360+0000 7ff286ff5700 1 -- 172.21.15.88:0/3054578727 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff29402e070 con 0x562d779ba860 2023-07-14T17:44:07.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.360+0000 7ff2b5680b80 10 client.15309 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:07.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.360+0000 7ff2b5680b80 10 client.15309 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T17:44:07.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.360+0000 7ff2b5680b80 20 client.15309 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:07.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.360+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/3054578727 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x562d77a27e00 0x562d77a247f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:07.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.360+0000 7ff2b5680b80 4 client.15309 mds_command: new command op to 24506 tid=0[{"prefix": "status"}] 2023-07-14T17:44:07.352 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.360+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3054578727 --> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562d77903e60 con 0x562d77a27e00 2023-07-14T17:44:07.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.360+0000 7ff2a2ffd700 1 --2- 172.21.15.88:0/3054578727 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x562d77a27e00 0x562d77a247f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:07.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.361+0000 7ff2a2ffd700 1 --2- 172.21.15.88:0/3054578727 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x562d77a27e00 0x562d77a247f0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=571374af2056a780 server_cookie=80af5be2eeb7c683 in_seq=0 out_seq=0 2023-07-14T17:44:07.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.361+0000 7ff286ff5700 10 client.15309 ms_handle_connect on v2:172.21.15.107:6838/31646548 2023-07-14T17:44:07.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.361+0000 7ff286ff5700 1 -- 172.21.15.88:0/3054578727 <== mds.0 v2:172.21.15.107:6838/31646548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x562d77903e60 con 0x562d77a27e00 2023-07-14T17:44:07.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.361+0000 7ff286ff5700 10 client.15309 handle_command_reply: tid=0 2023-07-14T17:44:07.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.361+0000 7ff2b5680b80 1 client.15309 shutdown 2023-07-14T17:44:07.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.361+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3054578727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 msgr2=0x562d779c6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.361+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/3054578727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 0x562d779c6cf0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7ff294031040 tx=0x7ff29400ff40 comp rx=0 tx=0).stop 2023-07-14T17:44:07.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.361+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3054578727 shutdown_connections 2023-07-14T17:44:07.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.361+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/3054578727 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x562d77a27e00 0x562d77a247f0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.361+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/3054578727 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x562d779bb460 0x562d779c9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.361+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/3054578727 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x562d779b9f10 0x562d779c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.361+0000 7ff2b5680b80 1 --2- 172.21.15.88:0/3054578727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x562d779ba860 0x562d779c6cf0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.361+0000 7ff286ff5700 0 client.15309 ms_handle_reset on v2:172.21.15.107:6838/31646548 2023-07-14T17:44:07.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.361+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3054578727 >> 172.21.15.88:0/3054578727 conn(0x562d779af630 msgr2=0x562d779c2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:07.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.361+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3054578727 shutdown_connections 2023-07-14T17:44:07.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.361+0000 7ff2b5680b80 1 -- 172.21.15.88:0/3054578727 wait complete. 2023-07-14T17:44:07.355 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.361+0000 7ff2b5680b80 20 client.15309 trim_cache size 0 max 16384 2023-07-14T17:44:07.356 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-07-14T17:44:07.356 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "b462a556-226c-11ee-9b33-001a4aab830c", 2023-07-14T17:44:07.356 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2023-07-14T17:44:07.356 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 24506, 2023-07-14T17:44:07.356 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2023-07-14T17:44:07.356 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2023-07-14T17:44:07.357 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2023-07-14T17:44:07.357 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 84.446086468999994, 2023-07-14T17:44:07.357 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 27, 2023-07-14T17:44:07.357 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 88, 2023-07-14T17:44:07.357 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 86, 2023-07-14T17:44:07.357 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 89.532430063999996 2023-07-14T17:44:07.357 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-07-14T17:44:07.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.362+0000 7ff2a94e8700 1 -- 172.21.15.88:0/2210241274 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff28c0a11e0 msgr2=0x7ff28c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.362+0000 7ff2a94e8700 1 --2- 172.21.15.88:0/2210241274 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff28c0a11e0 0x7ff28c0a3690 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff294005950 tx=0x7ff29400a300 comp rx=0 tx=0).stop 2023-07-14T17:44:07.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.362+0000 7ff2a94e8700 1 -- 172.21.15.88:0/2210241274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a40fdf20 msgr2=0x7ff2a4195870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:07.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.362+0000 7ff2a94e8700 1 --2- 172.21.15.88:0/2210241274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a40fdf20 0x7ff2a4195870 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff29800c010 tx=0x7ff298004730 comp rx=0 tx=0).stop 2023-07-14T17:44:07.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.362+0000 7ff2a94e8700 1 -- 172.21.15.88:0/2210241274 shutdown_connections 2023-07-14T17:44:07.359 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.362+0000 7ff2a94e8700 1 --2- 172.21.15.88:0/2210241274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff2a4101920 0x7ff2a419ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.362+0000 7ff2a94e8700 1 --2- 172.21.15.88:0/2210241274 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff28c0a11e0 0x7ff28c0a3690 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.362+0000 7ff2a94e8700 1 --2- 172.21.15.88:0/2210241274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff2a40fdf20 0x7ff2a4195870 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.362+0000 7ff2a94e8700 1 --2- 172.21.15.88:0/2210241274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff2a4100f90 0x7ff2a4195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:07.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.362+0000 7ff2a94e8700 1 -- 172.21.15.88:0/2210241274 >> 172.21.15.88:0/2210241274 conn(0x7ff2a40f98e0 msgr2=0x7ff2a41044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:07.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.362+0000 7ff2a94e8700 1 -- 172.21.15.88:0/2210241274 shutdown_connections 2023-07-14T17:44:07.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:07.362+0000 7ff2a94e8700 1 -- 172.21.15.88:0/2210241274 wait complete. 2023-07-14T17:44:07.361 INFO:tasks.cephadm:Setup complete, yielding 2023-07-14T17:44:07.361 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-14T17:44:07.376 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi088.front.sepia.ceph.com 2023-07-14T17:44:07.377 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- bash -c 'ceph orch status' 2023-07-14T17:44:07.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:06 smithi178 ceph-mon[129368]: pgmap v299: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2023-07-14T17:44:07.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:06 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/685600365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:07.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:06 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2242484567' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:07.926 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:44:09.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.161+0000 7f83973b4700 1 -- 172.21.15.88:0/2010767219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83900ffdb0 msgr2=0x7f8390100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:09.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.161+0000 7f83973b4700 1 --2- 172.21.15.88:0/2010767219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83900ffdb0 0x7f8390100200 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8380009990 tx=0x7f8380009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:09.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.161+0000 7f83973b4700 1 -- 172.21.15.88:0/2010767219 shutdown_connections 2023-07-14T17:44:09.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.161+0000 7f83973b4700 1 --2- 172.21.15.88:0/2010767219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8390100740 0x7f8390107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:09.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.161+0000 7f83973b4700 1 --2- 172.21.15.88:0/2010767219 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83900febb0 0x7f83900fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:09.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.161+0000 7f83973b4700 1 --2- 172.21.15.88:0/2010767219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83900ffdb0 0x7f8390100200 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:09.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.161+0000 7f83973b4700 1 -- 172.21.15.88:0/2010767219 >> 172.21.15.88:0/2010767219 conn(0x7f83900fa1e0 msgr2=0x7f83900fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:09.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.161+0000 7f83973b4700 1 -- 172.21.15.88:0/2010767219 shutdown_connections 2023-07-14T17:44:09.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.161+0000 7f83973b4700 1 -- 172.21.15.88:0/2010767219 wait complete. 2023-07-14T17:44:09.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.161+0000 7f83973b4700 1 Processor -- start 2023-07-14T17:44:09.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.161+0000 7f83973b4700 1 -- start start 2023-07-14T17:44:09.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f83973b4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83900febb0 0x7f8390195810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:09.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f83973b4700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83900ffdb0 0x7f8390195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:09.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f83973b4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8390100740 0x7f839019cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:09.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f83973b4700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839019d3a0 con 0x7f8390100740 2023-07-14T17:44:09.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f83973b4700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839019d4e0 con 0x7f83900febb0 2023-07-14T17:44:09.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f83973b4700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839019d670 con 0x7f83900ffdb0 2023-07-14T17:44:09.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f8395951700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8390100740 0x7f839019cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:09.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f8395951700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8390100740 0x7f839019cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42924/0 (socket says 172.21.15.88:42924) 2023-07-14T17:44:09.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f8395951700 1 -- 172.21.15.88:0/2185940550 learned_addr learned my addr 172.21.15.88:0/2185940550 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:09.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f839494f700 1 --2- 172.21.15.88:0/2185940550 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83900ffdb0 0x7f8390195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:09.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f8395150700 1 --2- 172.21.15.88:0/2185940550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83900febb0 0x7f8390195810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:09.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f8395951700 1 -- 172.21.15.88:0/2185940550 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83900ffdb0 msgr2=0x7f8390195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:09.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f8395951700 1 --2- 172.21.15.88:0/2185940550 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83900ffdb0 0x7f8390195d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:09.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f8395951700 1 -- 172.21.15.88:0/2185940550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83900febb0 msgr2=0x7f8390195810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:09.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f8395951700 1 --2- 172.21.15.88:0/2185940550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83900febb0 0x7f8390195810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:09.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f8395951700 1 -- 172.21.15.88:0/2185940550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83800095f0 con 0x7f8390100740 2023-07-14T17:44:09.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f8395150700 1 --2- 172.21.15.88:0/2185940550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83900febb0 0x7f8390195810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:09.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f8395951700 1 --2- 172.21.15.88:0/2185940550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8390100740 0x7f839019cdd0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8388007f10 tx=0x7f838800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:09.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f83867fc700 1 -- 172.21.15.88:0/2185940550 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8388017070 con 0x7f8390100740 2023-07-14T17:44:09.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f83867fc700 1 -- 172.21.15.88:0/2185940550 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f838800f040 con 0x7f8390100740 2023-07-14T17:44:09.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f83973b4700 1 -- 172.21.15.88:0/2185940550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839019d950 con 0x7f8390100740 2023-07-14T17:44:09.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.162+0000 7f83867fc700 1 -- 172.21.15.88:0/2185940550 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8388005710 con 0x7f8390100740 2023-07-14T17:44:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.163+0000 7f83973b4700 1 -- 172.21.15.88:0/2185940550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839019de00 con 0x7f8390100740 2023-07-14T17:44:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.163+0000 7f83973b4700 1 -- 172.21.15.88:0/2185940550 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f839010a220 con 0x7f8390100740 2023-07-14T17:44:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.164+0000 7f83867fc700 1 -- 172.21.15.88:0/2185940550 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8388004ad0 con 0x7f8390100740 2023-07-14T17:44:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.165+0000 7f83867fc700 1 --2- 172.21.15.88:0/2185940550 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f837c0a11e0 0x7f837c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.165+0000 7f8395150700 1 --2- 172.21.15.88:0/2185940550 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f837c0a11e0 0x7f837c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.165+0000 7f83867fc700 1 -- 172.21.15.88:0/2185940550 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f83880ccc10 con 0x7f8390100740 2023-07-14T17:44:09.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.165+0000 7f8395150700 1 --2- 172.21.15.88:0/2185940550 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f837c0a11e0 0x7f837c0a3690 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f83900ffc10 tx=0x7f838c00a680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:09.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.168+0000 7f83867fc700 1 -- 172.21.15.88:0/2185940550 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8388095ef0 con 0x7f8390100740 2023-07-14T17:44:09.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:08 smithi088 ceph-mon[127465]: pgmap v300: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2023-07-14T17:44:09.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:08 smithi107 ceph-mon[128999]: pgmap v300: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2023-07-14T17:44:09.353 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.372+0000 7f83973b4700 1 -- 172.21.15.88:0/2185940550 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8390068890 con 0x7f837c0a11e0 2023-07-14T17:44:09.354 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.373+0000 7f83867fc700 1 -- 172.21.15.88:0/2185940550 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f8390068890 con 0x7f837c0a11e0 2023-07-14T17:44:09.354 INFO:teuthology.orchestra.run.smithi088.stdout:Backend: cephadm 2023-07-14T17:44:09.354 INFO:teuthology.orchestra.run.smithi088.stdout:Available: Yes 2023-07-14T17:44:09.355 INFO:teuthology.orchestra.run.smithi088.stdout:Paused: No 2023-07-14T17:44:09.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.375+0000 7f83973b4700 1 -- 172.21.15.88:0/2185940550 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f837c0a11e0 msgr2=0x7f837c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:09.356 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.375+0000 7f83973b4700 1 --2- 172.21.15.88:0/2185940550 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f837c0a11e0 0x7f837c0a3690 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f83900ffc10 tx=0x7f838c00a680 comp rx=0 tx=0).stop 2023-07-14T17:44:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.375+0000 7f83973b4700 1 -- 172.21.15.88:0/2185940550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8390100740 msgr2=0x7f839019cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.375+0000 7f83973b4700 1 --2- 172.21.15.88:0/2185940550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8390100740 0x7f839019cdd0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8388007f10 tx=0x7f838800d3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.375+0000 7f83973b4700 1 -- 172.21.15.88:0/2185940550 shutdown_connections 2023-07-14T17:44:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.375+0000 7f83973b4700 1 --2- 172.21.15.88:0/2185940550 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f837c0a11e0 0x7f837c0a3690 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.375+0000 7f83973b4700 1 --2- 172.21.15.88:0/2185940550 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8390100740 0x7f839019cdd0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.375+0000 7f83973b4700 1 --2- 172.21.15.88:0/2185940550 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f83900febb0 0x7f8390195810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:09.357 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.375+0000 7f83973b4700 1 --2- 172.21.15.88:0/2185940550 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f83900ffdb0 0x7f8390195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:09.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.375+0000 7f83973b4700 1 -- 172.21.15.88:0/2185940550 >> 172.21.15.88:0/2185940550 conn(0x7f83900fa1e0 msgr2=0x7f8390068180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:09.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.375+0000 7f83973b4700 1 -- 172.21.15.88:0/2185940550 shutdown_connections 2023-07-14T17:44:09.358 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:09.375+0000 7f83973b4700 1 -- 172.21.15.88:0/2185940550 wait complete. 2023-07-14T17:44:09.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:08 smithi178 ceph-mon[129368]: pgmap v300: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2023-07-14T17:44:09.787 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- bash -c 'ceph orch ps' 2023-07-14T17:44:10.343 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:44:11.153 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:10 smithi088 ceph-mon[127465]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:44:11.153 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:10 smithi088 ceph-mon[127465]: pgmap v301: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T17:44:11.223 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:10 smithi107 ceph-mon[128999]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:44:11.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:10 smithi107 ceph-mon[128999]: pgmap v301: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T17:44:11.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:10 smithi178 ceph-mon[129368]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:44:11.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:10 smithi178 ceph-mon[129368]: pgmap v301: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T17:44:11.642 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.659+0000 7f56327dc700 1 -- 172.21.15.88:0/2639517971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f562c0fe8e0 msgr2=0x7f562c101010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:11.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.659+0000 7f56327dc700 1 --2- 172.21.15.88:0/2639517971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f562c0fe8e0 0x7f562c101010 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5620009990 tx=0x7f5620009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:11.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.659+0000 7f56327dc700 1 -- 172.21.15.88:0/2639517971 shutdown_connections 2023-07-14T17:44:11.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.659+0000 7f56327dc700 1 --2- 172.21.15.88:0/2639517971 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f562c1016a0 0x7f562c108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:11.643 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.659+0000 7f56327dc700 1 --2- 172.21.15.88:0/2639517971 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f562c0fdf90 0x7f562c0fe3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.659+0000 7f56327dc700 1 --2- 172.21.15.88:0/2639517971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f562c0fe8e0 0x7f562c101010 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.659+0000 7f56327dc700 1 -- 172.21.15.88:0/2639517971 >> 172.21.15.88:0/2639517971 conn(0x7f562c0f99c0 msgr2=0x7f562c0fbdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.659+0000 7f56327dc700 1 -- 172.21.15.88:0/2639517971 shutdown_connections 2023-07-14T17:44:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.659+0000 7f56327dc700 1 -- 172.21.15.88:0/2639517971 wait complete. 2023-07-14T17:44:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.660+0000 7f56327dc700 1 Processor -- start 2023-07-14T17:44:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.660+0000 7f56327dc700 1 -- start start 2023-07-14T17:44:11.644 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.660+0000 7f56327dc700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f562c0fdf90 0x7f562c195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.660+0000 7f56327dc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f562c0fe8e0 0x7f562c195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.660+0000 7f56327dc700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f562c1016a0 0x7f562c19cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.660+0000 7f56327dc700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c19d510 con 0x7f562c0fe8e0 2023-07-14T17:44:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.660+0000 7f56327dc700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c19d650 con 0x7f562c1016a0 2023-07-14T17:44:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.660+0000 7f56327dc700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f562c19d7e0 con 0x7f562c0fdf90 2023-07-14T17:44:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.660+0000 7f562b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f562c0fe8e0 0x7f562c195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.660+0000 7f562b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f562c0fe8e0 0x7f562c195ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42936/0 (socket says 172.21.15.88:42936) 2023-07-14T17:44:11.645 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.660+0000 7f562b7fe700 1 -- 172.21.15.88:0/775203274 learned_addr learned my addr 172.21.15.88:0/775203274 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.660+0000 7f562bfff700 1 --2- 172.21.15.88:0/775203274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f562c0fdf90 0x7f562c195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.661+0000 7f5630d79700 1 --2- 172.21.15.88:0/775203274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f562c1016a0 0x7f562c19cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.661+0000 7f562b7fe700 1 -- 172.21.15.88:0/775203274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f562c0fdf90 msgr2=0x7f562c195980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.661+0000 7f562b7fe700 1 --2- 172.21.15.88:0/775203274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f562c0fdf90 0x7f562c195980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.661+0000 7f562b7fe700 1 -- 172.21.15.88:0/775203274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f562c1016a0 msgr2=0x7f562c19cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.661+0000 7f562b7fe700 1 --2- 172.21.15.88:0/775203274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f562c1016a0 0x7f562c19cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:11.646 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.661+0000 7f562b7fe700 1 -- 172.21.15.88:0/775203274 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56200095f0 con 0x7f562c0fe8e0 2023-07-14T17:44:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.661+0000 7f5630d79700 1 --2- 172.21.15.88:0/775203274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f562c1016a0 0x7f562c19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.661+0000 7f562b7fe700 1 --2- 172.21.15.88:0/775203274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f562c0fe8e0 0x7f562c195ec0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5620003f60 tx=0x7f56200047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.661+0000 7f56297fa700 1 -- 172.21.15.88:0/775203274 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562001d070 con 0x7f562c0fe8e0 2023-07-14T17:44:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.661+0000 7f56297fa700 1 -- 172.21.15.88:0/775203274 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f562000be50 con 0x7f562c0fe8e0 2023-07-14T17:44:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.661+0000 7f56327dc700 1 -- 172.21.15.88:0/775203274 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f562c19da60 con 0x7f562c0fe8e0 2023-07-14T17:44:11.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.661+0000 7f56297fa700 1 -- 172.21.15.88:0/775203274 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562000f460 con 0x7f562c0fe8e0 2023-07-14T17:44:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.661+0000 7f56327dc700 1 -- 172.21.15.88:0/775203274 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f562c19de70 con 0x7f562c0fe8e0 2023-07-14T17:44:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.662+0000 7f56327dc700 1 -- 172.21.15.88:0/775203274 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f562c1074b0 con 0x7f562c0fe8e0 2023-07-14T17:44:11.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.663+0000 7f56297fa700 1 -- 172.21.15.88:0/775203274 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f56200226e0 con 0x7f562c0fe8e0 2023-07-14T17:44:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.664+0000 7f56297fa700 1 --2- 172.21.15.88:0/775203274 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f56140a11e0 0x7f56140a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.664+0000 7f562bfff700 1 --2- 172.21.15.88:0/775203274 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f56140a11e0 0x7f56140a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.664+0000 7f56297fa700 1 -- 172.21.15.88:0/775203274 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f56200ce6f0 con 0x7f562c0fe8e0 2023-07-14T17:44:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.664+0000 7f562bfff700 1 --2- 172.21.15.88:0/775203274 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f56140a11e0 0x7f56140a3690 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f561c005950 tx=0x7f561c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:11.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.667+0000 7f56297fa700 1 -- 172.21.15.88:0/775203274 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5620097a60 con 0x7f562c0fe8e0 2023-07-14T17:44:11.841 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.859+0000 7f56327dc700 1 -- 172.21.15.88:0/775203274 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f562c060a80 con 0x7f56140a11e0 2023-07-14T17:44:11.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.871+0000 7f56297fa700 1 -- 172.21.15.88:0/775203274 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f562c060a80 con 0x7f56140a11e0 2023-07-14T17:44:11.853 INFO:teuthology.orchestra.run.smithi088.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-14T17:44:11.853 INFO:teuthology.orchestra.run.smithi088.stdout:mds.a smithi088 running (96s) - 96s - - 18.0.0-4940-g81c9387d 219b180907a8 987cb7698269 2023-07-14T17:44:11.854 INFO:teuthology.orchestra.run.smithi088.stdout:mds.b smithi107 running (94s) - 93s 31.3M - 18.0.0-4940-g81c9387d 219b180907a8 d13fd2934eaa 2023-07-14T17:44:11.854 INFO:teuthology.orchestra.run.smithi088.stdout:mds.c smithi178 running (105s) - 105s 18.7M - 18.0.0-4940-g81c9387d 219b180907a8 215025bb1f80 2023-07-14T17:44:11.854 INFO:teuthology.orchestra.run.smithi088.stdout:mds.d smithi088 running (112s) - 112s - - 18.0.0-4940-g81c9387d 219b180907a8 ea6ec2155e12 2023-07-14T17:44:11.854 INFO:teuthology.orchestra.run.smithi088.stdout:mds.e smithi107 running (110s) - 110s 18.5M - 18.0.0-4940-g81c9387d 219b180907a8 0127cf452a76 2023-07-14T17:44:11.854 INFO:teuthology.orchestra.run.smithi088.stdout:mds.f smithi178 running (108s) - 107s 21.2M - 18.0.0-4940-g81c9387d 219b180907a8 1ca0ea4f0d31 2023-07-14T17:44:11.854 INFO:teuthology.orchestra.run.smithi088.stdout:mds.g smithi088 running (99s) - 98s - - 18.0.0-4940-g81c9387d 219b180907a8 6676d3bfb7e5 2023-07-14T17:44:11.854 INFO:teuthology.orchestra.run.smithi088.stdout:mds.h smithi107 running (103s) - 103s 18.7M - 18.0.0-4940-g81c9387d 219b180907a8 b81cdf5c404c 2023-07-14T17:44:11.855 INFO:teuthology.orchestra.run.smithi088.stdout:mds.i smithi178 running (101s) - 100s 21.1M - 18.0.0-4940-g81c9387d 219b180907a8 57ec7a569420 2023-07-14T17:44:11.855 INFO:teuthology.orchestra.run.smithi088.stdout:mds.j smithi088 running (114s) - 114s - - 18.0.0-4940-g81c9387d 219b180907a8 b69aea97d5e5 2023-07-14T17:44:11.855 INFO:teuthology.orchestra.run.smithi088.stdout:mds.k smithi107 running (116s) - 116s 17.4M - 18.0.0-4940-g81c9387d 219b180907a8 8f79948b4e4d 2023-07-14T17:44:11.855 INFO:teuthology.orchestra.run.smithi088.stdout:mds.l smithi178 running (119s) - 118s 17.5M - 18.0.0-4940-g81c9387d 219b180907a8 e3c2b8f1de44 2023-07-14T17:44:11.855 INFO:teuthology.orchestra.run.smithi088.stdout:mgr.x smithi088 *:9283,8765 running (8m) - 8m - - 18.0.0-4940-g81c9387d 219b180907a8 7b3ece0f3903 2023-07-14T17:44:11.855 INFO:teuthology.orchestra.run.smithi088.stdout:mgr.y smithi107 *:8443,8765 running (6m) - 6m 446M - 18.0.0-4940-g81c9387d 219b180907a8 e740b1f28d2a 2023-07-14T17:44:11.855 INFO:teuthology.orchestra.run.smithi088.stdout:mgr.z smithi178 *:8443,8765 running (6m) - 6m 446M - 18.0.0-4940-g81c9387d 219b180907a8 47bbe9518e20 2023-07-14T17:44:11.855 INFO:teuthology.orchestra.run.smithi088.stdout:mon.a smithi088 running (8m) - 8m - 2048M 18.0.0-4940-g81c9387d 219b180907a8 308edc0019a3 2023-07-14T17:44:11.856 INFO:teuthology.orchestra.run.smithi088.stdout:mon.b smithi107 running (6m) - 6m 63.6M 2048M 18.0.0-4940-g81c9387d 219b180907a8 db1219f65df0 2023-07-14T17:44:11.856 INFO:teuthology.orchestra.run.smithi088.stdout:mon.c smithi178 running (6m) - 6m 59.6M 2048M 18.0.0-4940-g81c9387d 219b180907a8 f46f8e6951c9 2023-07-14T17:44:11.856 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0 smithi088 running (6m) - 6m - 4292M 18.0.0-4940-g81c9387d 219b180907a8 5b5f3d03c0de 2023-07-14T17:44:11.856 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1 smithi107 running (5m) - 5m 77.3M 4292M 18.0.0-4940-g81c9387d 219b180907a8 3290e5daa0a1 2023-07-14T17:44:11.856 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2 smithi178 running (5m) - 5m 78.2M 4292M 18.0.0-4940-g81c9387d 219b180907a8 8ef9ab4539a1 2023-07-14T17:44:11.856 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3 smithi088 running (5m) - 5m - 4292M 18.0.0-4940-g81c9387d 219b180907a8 937645c367ec 2023-07-14T17:44:11.856 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4 smithi107 running (4m) - 4m 66.6M 4292M 18.0.0-4940-g81c9387d 219b180907a8 2df975096bb0 2023-07-14T17:44:11.857 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5 smithi178 running (4m) - 4m 68.4M 4292M 18.0.0-4940-g81c9387d 219b180907a8 361cd43da0f6 2023-07-14T17:44:11.857 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6 smithi088 running (4m) - 4m - 4292M 18.0.0-4940-g81c9387d 219b180907a8 4d38f1ba5256 2023-07-14T17:44:11.857 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7 smithi107 running (3m) - 3m 68.3M 4292M 18.0.0-4940-g81c9387d 219b180907a8 4e028fd6649d 2023-07-14T17:44:11.857 INFO:teuthology.orchestra.run.smithi088.stdout:osd.8 smithi178 running (3m) - 3m 63.1M 4292M 18.0.0-4940-g81c9387d 219b180907a8 ce136f51f8a8 2023-07-14T17:44:11.857 INFO:teuthology.orchestra.run.smithi088.stdout:osd.9 smithi088 running (3m) - 2m - 4292M 18.0.0-4940-g81c9387d 219b180907a8 5ba592aab093 2023-07-14T17:44:11.857 INFO:teuthology.orchestra.run.smithi088.stdout:osd.10 smithi107 running (2m) - 2m 65.7M 4292M 18.0.0-4940-g81c9387d 219b180907a8 7e92e35ad932 2023-07-14T17:44:11.858 INFO:teuthology.orchestra.run.smithi088.stdout:osd.11 smithi178 running (2m) - 2m 56.7M 4292M 18.0.0-4940-g81c9387d 219b180907a8 0faa08da3ec3 2023-07-14T17:44:11.858 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.874+0000 7f56327dc700 1 -- 172.21.15.88:0/775203274 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f56140a11e0 msgr2=0x7f56140a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:11.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.874+0000 7f56327dc700 1 --2- 172.21.15.88:0/775203274 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f56140a11e0 0x7f56140a3690 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f561c005950 tx=0x7f561c00b410 comp rx=0 tx=0).stop 2023-07-14T17:44:11.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.874+0000 7f56327dc700 1 -- 172.21.15.88:0/775203274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f562c0fe8e0 msgr2=0x7f562c195ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:11.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.874+0000 7f56327dc700 1 --2- 172.21.15.88:0/775203274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f562c0fe8e0 0x7f562c195ec0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5620003f60 tx=0x7f56200047a0 comp rx=0 tx=0).stop 2023-07-14T17:44:11.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.874+0000 7f56327dc700 1 -- 172.21.15.88:0/775203274 shutdown_connections 2023-07-14T17:44:11.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.874+0000 7f56327dc700 1 --2- 172.21.15.88:0/775203274 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f562c1016a0 0x7f562c19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:11.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.874+0000 7f56327dc700 1 --2- 172.21.15.88:0/775203274 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f562c0fdf90 0x7f562c195980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:11.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.874+0000 7f56327dc700 1 --2- 172.21.15.88:0/775203274 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f56140a11e0 0x7f56140a3690 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:11.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.874+0000 7f56327dc700 1 --2- 172.21.15.88:0/775203274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f562c0fe8e0 0x7f562c195ec0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:11.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.874+0000 7f56327dc700 1 -- 172.21.15.88:0/775203274 >> 172.21.15.88:0/775203274 conn(0x7f562c0f99c0 msgr2=0x7f562c1045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:11.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.874+0000 7f56327dc700 1 -- 172.21.15.88:0/775203274 shutdown_connections 2023-07-14T17:44:11.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:11.874+0000 7f56327dc700 1 -- 172.21.15.88:0/775203274 wait complete. 2023-07-14T17:44:12.191 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- bash -c 'ceph orch ls' 2023-07-14T17:44:12.744 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:44:13.223 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:12 smithi107 ceph-mon[128999]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:44:13.224 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:12 smithi107 ceph-mon[128999]: pgmap v302: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2023-07-14T17:44:13.321 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:12 smithi088 ceph-mon[127465]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:44:13.322 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:12 smithi088 ceph-mon[127465]: pgmap v302: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2023-07-14T17:44:13.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:12 smithi178 ceph-mon[129368]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:44:13.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:12 smithi178 ceph-mon[129368]: pgmap v302: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2023-07-14T17:44:14.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.125+0000 7f42f9632700 1 -- 172.21.15.88:0/3080104540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42f4100f40 msgr2=0x7f42f4101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:14.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.125+0000 7f42f9632700 1 --2- 172.21.15.88:0/3080104540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42f4100f40 0x7f42f4101390 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f42e4009990 tx=0x7f42e4009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:14.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.125+0000 7f42f9632700 1 -- 172.21.15.88:0/3080104540 shutdown_connections 2023-07-14T17:44:14.110 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.125+0000 7f42f9632700 1 --2- 172.21.15.88:0/3080104540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42f41018d0 0x7f42f4108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:14.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.125+0000 7f42f9632700 1 --2- 172.21.15.88:0/3080104540 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f42f40ffd40 0x7f42f4100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:14.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.125+0000 7f42f9632700 1 --2- 172.21.15.88:0/3080104540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42f4100f40 0x7f42f4101390 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:14.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.125+0000 7f42f9632700 1 -- 172.21.15.88:0/3080104540 >> 172.21.15.88:0/3080104540 conn(0x7f42f40fb350 msgr2=0x7f42f40fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:14.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.125+0000 7f42f9632700 1 -- 172.21.15.88:0/3080104540 shutdown_connections 2023-07-14T17:44:14.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.126+0000 7f42f9632700 1 -- 172.21.15.88:0/3080104540 wait complete. 2023-07-14T17:44:14.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.126+0000 7f42f9632700 1 Processor -- start 2023-07-14T17:44:14.111 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.126+0000 7f42f9632700 1 -- start start 2023-07-14T17:44:14.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f9632700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42f40ffd40 0x7f42f41958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:14.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f9632700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f42f4100f40 0x7f42f4195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:14.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f9632700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42f41018d0 0x7f42f419ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:14.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f9632700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f419d460 con 0x7f42f41018d0 2023-07-14T17:44:14.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f9632700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f419d5a0 con 0x7f42f40ffd40 2023-07-14T17:44:14.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f9632700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f419d730 con 0x7f42f4100f40 2023-07-14T17:44:14.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f37fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42f41018d0 0x7f42f419ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:14.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f37fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42f41018d0 0x7f42f419ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42952/0 (socket says 172.21.15.88:42952) 2023-07-14T17:44:14.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f37fe700 1 -- 172.21.15.88:0/111341348 learned_addr learned my addr 172.21.15.88:0/111341348 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:14.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f2ffd700 1 --2- 172.21.15.88:0/111341348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42f40ffd40 0x7f42f41958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:14.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f27fc700 1 --2- 172.21.15.88:0/111341348 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f42f4100f40 0x7f42f4195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:14.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f37fe700 1 -- 172.21.15.88:0/111341348 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f42f4100f40 msgr2=0x7f42f4195e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:14.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f37fe700 1 --2- 172.21.15.88:0/111341348 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f42f4100f40 0x7f42f4195e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:14.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f37fe700 1 -- 172.21.15.88:0/111341348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42f40ffd40 msgr2=0x7f42f41958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:14.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f37fe700 1 --2- 172.21.15.88:0/111341348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42f40ffd40 0x7f42f41958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:14.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f37fe700 1 -- 172.21.15.88:0/111341348 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e40095f0 con 0x7f42f41018d0 2023-07-14T17:44:14.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f27fc700 1 --2- 172.21.15.88:0/111341348 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f42f4100f40 0x7f42f4195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:14.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f2ffd700 1 --2- 172.21.15.88:0/111341348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42f40ffd40 0x7f42f41958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:14.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f37fe700 1 --2- 172.21.15.88:0/111341348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42f41018d0 0x7f42f419ce90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00ef60 tx=0x7f42ec00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:14.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42ebfff700 1 -- 172.21.15.88:0/111341348 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec018070 con 0x7f42f41018d0 2023-07-14T17:44:14.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42ebfff700 1 -- 172.21.15.88:0/111341348 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f42ec00f040 con 0x7f42f41018d0 2023-07-14T17:44:14.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f9632700 1 -- 172.21.15.88:0/111341348 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f419d9b0 con 0x7f42f41018d0 2023-07-14T17:44:14.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42ebfff700 1 -- 172.21.15.88:0/111341348 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42ec003d30 con 0x7f42f41018d0 2023-07-14T17:44:14.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.127+0000 7f42f9632700 1 -- 172.21.15.88:0/111341348 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f419de60 con 0x7f42f41018d0 2023-07-14T17:44:14.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.128+0000 7f42f9632700 1 -- 172.21.15.88:0/111341348 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f4107410 con 0x7f42f41018d0 2023-07-14T17:44:14.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.130+0000 7f42ebfff700 1 -- 172.21.15.88:0/111341348 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f42ec004ad0 con 0x7f42f41018d0 2023-07-14T17:44:14.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.130+0000 7f42ebfff700 1 --2- 172.21.15.88:0/111341348 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f42e00a11e0 0x7f42e00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:14.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.130+0000 7f42f2ffd700 1 --2- 172.21.15.88:0/111341348 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f42e00a11e0 0x7f42e00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:14.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.130+0000 7f42ebfff700 1 -- 172.21.15.88:0/111341348 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f42ec0cd2e0 con 0x7f42f41018d0 2023-07-14T17:44:14.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.131+0000 7f42f2ffd700 1 --2- 172.21.15.88:0/111341348 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f42e00a11e0 0x7f42e00a3690 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f42dc005950 tx=0x7f42dc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:14.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.134+0000 7f42ebfff700 1 -- 172.21.15.88:0/111341348 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f42ec0966f0 con 0x7f42f41018d0 2023-07-14T17:44:14.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.328+0000 7f42f9632700 1 -- 172.21.15.88:0/111341348 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f42f4060a80 con 0x7f42e00a11e0 2023-07-14T17:44:14.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.333+0000 7f42ebfff700 1 -- 172.21.15.88:0/111341348 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f42f4060a80 con 0x7f42e00a11e0 2023-07-14T17:44:14.316 INFO:teuthology.orchestra.run.smithi088.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-07-14T17:44:14.316 INFO:teuthology.orchestra.run.smithi088.stdout:mds.all 12/12 - 2m smithi088=a;smithi088=d;smithi088=g;smithi088=j;smithi107=b;smithi107=e;smithi107=h;smithi107=k;smithi178=c;smithi178=f;smithi178=i;smithi178=l;count:12 2023-07-14T17:44:14.316 INFO:teuthology.orchestra.run.smithi088.stdout:mgr 3/3 - 6m smithi088=x;smithi107=y;smithi178=z;count:3 2023-07-14T17:44:14.317 INFO:teuthology.orchestra.run.smithi088.stdout:mon 3/3 - 6m smithi088:172.21.15.88=a;smithi107:172.21.15.107=b;smithi178:172.21.15.178=c;count:3 2023-07-14T17:44:14.317 INFO:teuthology.orchestra.run.smithi088.stdout:osd 12 - - 2023-07-14T17:44:14.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.335+0000 7f42f9632700 1 -- 172.21.15.88:0/111341348 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f42e00a11e0 msgr2=0x7f42e00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:14.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.335+0000 7f42f9632700 1 --2- 172.21.15.88:0/111341348 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f42e00a11e0 0x7f42e00a3690 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f42dc005950 tx=0x7f42dc0058e0 comp rx=0 tx=0).stop 2023-07-14T17:44:14.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.335+0000 7f42f9632700 1 -- 172.21.15.88:0/111341348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42f41018d0 msgr2=0x7f42f419ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:14.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.335+0000 7f42f9632700 1 --2- 172.21.15.88:0/111341348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42f41018d0 0x7f42f419ce90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00ef60 tx=0x7f42ec00ef90 comp rx=0 tx=0).stop 2023-07-14T17:44:14.318 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.335+0000 7f42f9632700 1 -- 172.21.15.88:0/111341348 shutdown_connections 2023-07-14T17:44:14.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.335+0000 7f42f9632700 1 --2- 172.21.15.88:0/111341348 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f42e00a11e0 0x7f42e00a3690 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:14.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.335+0000 7f42f9632700 1 --2- 172.21.15.88:0/111341348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42f41018d0 0x7f42f419ce90 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:14.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.335+0000 7f42f9632700 1 --2- 172.21.15.88:0/111341348 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f42f40ffd40 0x7f42f41958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:14.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.335+0000 7f42f9632700 1 --2- 172.21.15.88:0/111341348 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f42f4100f40 0x7f42f4195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:14.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.335+0000 7f42f9632700 1 -- 172.21.15.88:0/111341348 >> 172.21.15.88:0/111341348 conn(0x7f42f40fb350 msgr2=0x7f42f4104550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:14.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.335+0000 7f42f9632700 1 -- 172.21.15.88:0/111341348 shutdown_connections 2023-07-14T17:44:14.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:14.335+0000 7f42f9632700 1 -- 172.21.15.88:0/111341348 wait complete. 2023-07-14T17:44:14.898 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- bash -c 'ceph orch host ls' 2023-07-14T17:44:15.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:14 smithi088 ceph-mon[127465]: pgmap v303: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T17:44:15.223 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:14 smithi107 ceph-mon[128999]: pgmap v303: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T17:44:15.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:14 smithi178 ceph-mon[129368]: pgmap v303: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T17:44:15.456 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:44:16.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:15 smithi178 ceph-mon[129368]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:44:16.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:15 smithi178 ceph-mon[129368]: mgrmap e18: x(active, since 7m), standbys: z, y 2023-07-14T17:44:16.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:15 smithi088 ceph-mon[127465]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:44:16.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:15 smithi088 ceph-mon[127465]: mgrmap e18: x(active, since 7m), standbys: z, y 2023-07-14T17:44:16.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:15 smithi107 ceph-mon[128999]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:44:16.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:15 smithi107 ceph-mon[128999]: mgrmap e18: x(active, since 7m), standbys: z, y 2023-07-14T17:44:16.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.675+0000 7f464c22f700 1 -- 172.21.15.88:0/392039850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4644100820 msgr2=0x7f46441070c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:16.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.675+0000 7f464c22f700 1 --2- 172.21.15.88:0/392039850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4644100820 0x7f46441070c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f463c00b0a0 tx=0x7f463c00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:16.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.675+0000 7f464c22f700 1 -- 172.21.15.88:0/392039850 shutdown_connections 2023-07-14T17:44:16.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.675+0000 7f464c22f700 1 --2- 172.21.15.88:0/392039850 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4644100820 0x7f46441070c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:16.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.675+0000 7f464c22f700 1 --2- 172.21.15.88:0/392039850 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46440febd0 0x7f46440fefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:16.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.675+0000 7f464c22f700 1 --2- 172.21.15.88:0/392039850 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46440ffe70 0x7f46441002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:16.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.675+0000 7f464c22f700 1 -- 172.21.15.88:0/392039850 >> 172.21.15.88:0/392039850 conn(0x7f46440fa240 msgr2=0x7f46440fc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:16.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.676+0000 7f464c22f700 1 -- 172.21.15.88:0/392039850 shutdown_connections 2023-07-14T17:44:16.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.676+0000 7f464c22f700 1 -- 172.21.15.88:0/392039850 wait complete. 2023-07-14T17:44:16.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.676+0000 7f464c22f700 1 Processor -- start 2023-07-14T17:44:16.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.676+0000 7f464c22f700 1 -- start start 2023-07-14T17:44:16.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.676+0000 7f464c22f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46440febd0 0x7f4644102c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:16.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.676+0000 7f464c22f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46440ffe70 0x7f4644101250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:16.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.676+0000 7f464c22f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4644100820 0x7f4644101790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:16.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.676+0000 7f464c22f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4644101f10 con 0x7f4644100820 2023-07-14T17:44:16.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.676+0000 7f464c22f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4644102080 con 0x7f46440febd0 2023-07-14T17:44:16.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.676+0000 7f464c22f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464419fb20 con 0x7f46440ffe70 2023-07-14T17:44:16.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.677+0000 7f464a7cc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4644100820 0x7f4644101790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:16.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.677+0000 7f464a7cc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4644100820 0x7f4644101790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42962/0 (socket says 172.21.15.88:42962) 2023-07-14T17:44:16.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.677+0000 7f464a7cc700 1 -- 172.21.15.88:0/2994700064 learned_addr learned my addr 172.21.15.88:0/2994700064 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:16.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.677+0000 7f4649fcb700 1 --2- 172.21.15.88:0/2994700064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46440febd0 0x7f4644102c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:16.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.677+0000 7f46497ca700 1 --2- 172.21.15.88:0/2994700064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46440ffe70 0x7f4644101250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:16.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.677+0000 7f464a7cc700 1 -- 172.21.15.88:0/2994700064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46440ffe70 msgr2=0x7f4644101250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:16.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.677+0000 7f464a7cc700 1 --2- 172.21.15.88:0/2994700064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46440ffe70 0x7f4644101250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:16.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.677+0000 7f464a7cc700 1 -- 172.21.15.88:0/2994700064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46440febd0 msgr2=0x7f4644102c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:16.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.677+0000 7f464a7cc700 1 --2- 172.21.15.88:0/2994700064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46440febd0 0x7f4644102c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:16.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.677+0000 7f464a7cc700 1 -- 172.21.15.88:0/2994700064 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c009d00 con 0x7f4644100820 2023-07-14T17:44:16.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.677+0000 7f46497ca700 1 --2- 172.21.15.88:0/2994700064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46440ffe70 0x7f4644101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:16.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.677+0000 7f464a7cc700 1 --2- 172.21.15.88:0/2994700064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4644100820 0x7f4644101790 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f463c00b070 tx=0x7f463c004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:16.664 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.677+0000 7f4636ffd700 1 -- 172.21.15.88:0/2994700064 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c00e040 con 0x7f4644100820 2023-07-14T17:44:16.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.677+0000 7f4636ffd700 1 -- 172.21.15.88:0/2994700064 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f463c01bcf0 con 0x7f4644100820 2023-07-14T17:44:16.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.677+0000 7f464c22f700 1 -- 172.21.15.88:0/2994700064 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f464419fda0 con 0x7f4644100820 2023-07-14T17:44:16.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.677+0000 7f4636ffd700 1 -- 172.21.15.88:0/2994700064 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c0058d0 con 0x7f4644100820 2023-07-14T17:44:16.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.677+0000 7f464c22f700 1 -- 172.21.15.88:0/2994700064 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46441a0280 con 0x7f4644100820 2023-07-14T17:44:16.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.678+0000 7f464c22f700 1 -- 172.21.15.88:0/2994700064 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f464410a2d0 con 0x7f4644100820 2023-07-14T17:44:16.665 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.680+0000 7f4636ffd700 1 -- 172.21.15.88:0/2994700064 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f463c01b860 con 0x7f4644100820 2023-07-14T17:44:16.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.680+0000 7f4636ffd700 1 --2- 172.21.15.88:0/2994700064 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f46300a11d0 0x7f46300a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:16.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.680+0000 7f4649fcb700 1 --2- 172.21.15.88:0/2994700064 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f46300a11d0 0x7f46300a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:16.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.680+0000 7f4636ffd700 1 -- 172.21.15.88:0/2994700064 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f463c0cf1a0 con 0x7f4644100820 2023-07-14T17:44:16.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.681+0000 7f4649fcb700 1 --2- 172.21.15.88:0/2994700064 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f46300a11d0 0x7f46300a3680 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4640005fd0 tx=0x7f464000a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:16.666 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.683+0000 7f4636ffd700 1 -- 172.21.15.88:0/2994700064 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f463c098670 con 0x7f4644100820 2023-07-14T17:44:16.859 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.877+0000 7f464c22f700 1 -- 172.21.15.88:0/2994700064 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f46440691d0 con 0x7f46300a11d0 2023-07-14T17:44:16.860 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.878+0000 7f4636ffd700 1 -- 172.21.15.88:0/2994700064 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f46440691d0 con 0x7f46300a11d0 2023-07-14T17:44:16.861 INFO:teuthology.orchestra.run.smithi088.stdout:HOST ADDR LABELS STATUS 2023-07-14T17:44:16.861 INFO:teuthology.orchestra.run.smithi088.stdout:smithi088 172.21.15.88 2023-07-14T17:44:16.861 INFO:teuthology.orchestra.run.smithi088.stdout:smithi107 172.21.15.107 2023-07-14T17:44:16.861 INFO:teuthology.orchestra.run.smithi088.stdout:smithi178 172.21.15.178 2023-07-14T17:44:16.861 INFO:teuthology.orchestra.run.smithi088.stdout:3 hosts in cluster 2023-07-14T17:44:16.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.880+0000 7f464c22f700 1 -- 172.21.15.88:0/2994700064 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f46300a11d0 msgr2=0x7f46300a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:16.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.880+0000 7f464c22f700 1 --2- 172.21.15.88:0/2994700064 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f46300a11d0 0x7f46300a3680 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f4640005fd0 tx=0x7f464000a560 comp rx=0 tx=0).stop 2023-07-14T17:44:16.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.880+0000 7f464c22f700 1 -- 172.21.15.88:0/2994700064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4644100820 msgr2=0x7f4644101790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:16.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.880+0000 7f464c22f700 1 --2- 172.21.15.88:0/2994700064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4644100820 0x7f4644101790 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f463c00b070 tx=0x7f463c004db0 comp rx=0 tx=0).stop 2023-07-14T17:44:16.863 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.880+0000 7f464c22f700 1 -- 172.21.15.88:0/2994700064 shutdown_connections 2023-07-14T17:44:16.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.880+0000 7f464c22f700 1 --2- 172.21.15.88:0/2994700064 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f46300a11d0 0x7f46300a3680 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:16.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.880+0000 7f464c22f700 1 --2- 172.21.15.88:0/2994700064 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4644100820 0x7f4644101790 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:16.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.880+0000 7f464c22f700 1 --2- 172.21.15.88:0/2994700064 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f46440febd0 0x7f4644102c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:16.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.880+0000 7f464c22f700 1 --2- 172.21.15.88:0/2994700064 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f46440ffe70 0x7f4644101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:16.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.880+0000 7f464c22f700 1 -- 172.21.15.88:0/2994700064 >> 172.21.15.88:0/2994700064 conn(0x7f46440fa240 msgr2=0x7f4644068ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:16.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.880+0000 7f464c22f700 1 -- 172.21.15.88:0/2994700064 shutdown_connections 2023-07-14T17:44:16.864 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:16.880+0000 7f464c22f700 1 -- 172.21.15.88:0/2994700064 wait complete. 2023-07-14T17:44:17.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:16 smithi088 ceph-mon[127465]: pgmap v304: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 511 B/s wr, 11 op/s 2023-07-14T17:44:17.205 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- bash -c 'ceph orch device ls' 2023-07-14T17:44:17.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:16 smithi178 ceph-mon[129368]: pgmap v304: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 511 B/s wr, 11 op/s 2023-07-14T17:44:17.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:16 smithi107 ceph-mon[128999]: pgmap v304: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 511 B/s wr, 11 op/s 2023-07-14T17:44:17.758 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:44:18.262 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:17 smithi088 ceph-mon[127465]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:44:18.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:17 smithi178 ceph-mon[129368]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:44:18.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:17 smithi107 ceph-mon[128999]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:44:18.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.010+0000 7f3a4836c700 1 -- 172.21.15.88:0/4222089005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a400fec40 msgr2=0x7f3a400ff050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:18.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.010+0000 7f3a4836c700 1 --2- 172.21.15.88:0/4222089005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a400fec40 0x7f3a400ff050 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3a34009990 tx=0x7f3a34009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:18.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.010+0000 7f3a4836c700 1 -- 172.21.15.88:0/4222089005 shutdown_connections 2023-07-14T17:44:18.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.010+0000 7f3a4836c700 1 --2- 172.21.15.88:0/4222089005 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a401007d0 0x7f3a40107090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:18.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.010+0000 7f3a4836c700 1 --2- 172.21.15.88:0/4222089005 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a400fec40 0x7f3a400ff050 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:18.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.010+0000 7f3a4836c700 1 --2- 172.21.15.88:0/4222089005 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a400ffe40 0x7f3a40100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:18.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.010+0000 7f3a4836c700 1 -- 172.21.15.88:0/4222089005 >> 172.21.15.88:0/4222089005 conn(0x7f3a400fa290 msgr2=0x7f3a400fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:18.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.010+0000 7f3a4836c700 1 -- 172.21.15.88:0/4222089005 shutdown_connections 2023-07-14T17:44:18.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.010+0000 7f3a4836c700 1 -- 172.21.15.88:0/4222089005 wait complete. 2023-07-14T17:44:18.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.011+0000 7f3a4836c700 1 Processor -- start 2023-07-14T17:44:18.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.011+0000 7f3a4836c700 1 -- start start 2023-07-14T17:44:18.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.011+0000 7f3a4836c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a400fec40 0x7f3a40194820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:18.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.011+0000 7f3a4836c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a400ffe40 0x7f3a40194d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:18.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.011+0000 7f3a4836c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a401007d0 0x7f3a4019bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:18.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.011+0000 7f3a4836c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4019c3b0 con 0x7f3a401007d0 2023-07-14T17:44:18.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.011+0000 7f3a4836c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4019c4f0 con 0x7f3a400fec40 2023-07-14T17:44:18.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.011+0000 7f3a4836c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4019c680 con 0x7f3a400ffe40 2023-07-14T17:44:18.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.012+0000 7f3a46909700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a401007d0 0x7f3a4019bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:18.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.012+0000 7f3a46909700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a401007d0 0x7f3a4019bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42974/0 (socket says 172.21.15.88:42974) 2023-07-14T17:44:18.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.012+0000 7f3a46909700 1 -- 172.21.15.88:0/1670459941 learned_addr learned my addr 172.21.15.88:0/1670459941 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:18.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.012+0000 7f3a45907700 1 --2- 172.21.15.88:0/1670459941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a400ffe40 0x7f3a40194d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:18.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.012+0000 7f3a46108700 1 --2- 172.21.15.88:0/1670459941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a400fec40 0x7f3a40194820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:18.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.012+0000 7f3a46909700 1 -- 172.21.15.88:0/1670459941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a400ffe40 msgr2=0x7f3a40194d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:18.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.012+0000 7f3a46909700 1 --2- 172.21.15.88:0/1670459941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a400ffe40 0x7f3a40194d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:18.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.012+0000 7f3a46909700 1 -- 172.21.15.88:0/1670459941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a400fec40 msgr2=0x7f3a40194820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:18.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.012+0000 7f3a46909700 1 --2- 172.21.15.88:0/1670459941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a400fec40 0x7f3a40194820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:18.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.012+0000 7f3a46909700 1 -- 172.21.15.88:0/1670459941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a340095f0 con 0x7f3a401007d0 2023-07-14T17:44:19.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.012+0000 7f3a46909700 1 --2- 172.21.15.88:0/1670459941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a401007d0 0x7f3a4019bde0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3a3800b9a0 tx=0x7f3a3800bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:19.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.012+0000 7f3a337fe700 1 -- 172.21.15.88:0/1670459941 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3800d850 con 0x7f3a401007d0 2023-07-14T17:44:19.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.012+0000 7f3a337fe700 1 -- 172.21.15.88:0/1670459941 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3a3800de40 con 0x7f3a401007d0 2023-07-14T17:44:19.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.012+0000 7f3a337fe700 1 -- 172.21.15.88:0/1670459941 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a38017400 con 0x7f3a401007d0 2023-07-14T17:44:19.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.012+0000 7f3a4836c700 1 -- 172.21.15.88:0/1670459941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4019c960 con 0x7f3a401007d0 2023-07-14T17:44:19.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.012+0000 7f3a4836c700 1 -- 172.21.15.88:0/1670459941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4019ce90 con 0x7f3a401007d0 2023-07-14T17:44:19.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.013+0000 7f3a4836c700 1 -- 172.21.15.88:0/1670459941 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a40105c30 con 0x7f3a401007d0 2023-07-14T17:44:19.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.015+0000 7f3a337fe700 1 -- 172.21.15.88:0/1670459941 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3a3800d9b0 con 0x7f3a401007d0 2023-07-14T17:44:19.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.015+0000 7f3a337fe700 1 --2- 172.21.15.88:0/1670459941 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3a2c0a11d0 0x7f3a2c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:19.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.016+0000 7f3a46108700 1 --2- 172.21.15.88:0/1670459941 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3a2c0a11d0 0x7f3a2c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:19.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.016+0000 7f3a337fe700 1 -- 172.21.15.88:0/1670459941 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3a38006e40 con 0x7f3a401007d0 2023-07-14T17:44:19.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.016+0000 7f3a46108700 1 --2- 172.21.15.88:0/1670459941 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3a2c0a11d0 0x7f3a2c0a3680 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400b5c0 tx=0x7f3a34005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:19.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.019+0000 7f3a337fe700 1 -- 172.21.15.88:0/1670459941 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3a38095da0 con 0x7f3a401007d0 2023-07-14T17:44:19.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:18 smithi088 ceph-mon[127465]: pgmap v305: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 511 B/s wr, 9 op/s 2023-07-14T17:44:19.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.211+0000 7f3a4836c700 1 -- 172.21.15.88:0/1670459941 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a40060a80 con 0x7f3a2c0a11d0 2023-07-14T17:44:19.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.213+0000 7f3a337fe700 1 -- 172.21.15.88:0/1670459941 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7f3a40060a80 con 0x7f3a2c0a11d0 2023-07-14T17:44:19.196 INFO:teuthology.orchestra.run.smithi088.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-07-14T17:44:19.196 INFO:teuthology.orchestra.run.smithi088.stdout:smithi088 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400XD400BGN 372G No 3m ago LVM detected, locked 2023-07-14T17:44:19.196 INFO:teuthology.orchestra.run.smithi088.stdout:smithi107 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NK400BGN 372G No 2m ago LVM detected, locked 2023-07-14T17:44:19.196 INFO:teuthology.orchestra.run.smithi088.stdout:smithi178 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB75130029480DGN 447G No 2m ago LVM detected, locked 2023-07-14T17:44:19.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.215+0000 7f3a4836c700 1 -- 172.21.15.88:0/1670459941 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3a2c0a11d0 msgr2=0x7f3a2c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:19.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.215+0000 7f3a4836c700 1 --2- 172.21.15.88:0/1670459941 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3a2c0a11d0 0x7f3a2c0a3680 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400b5c0 tx=0x7f3a34005fb0 comp rx=0 tx=0).stop 2023-07-14T17:44:19.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.215+0000 7f3a4836c700 1 -- 172.21.15.88:0/1670459941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a401007d0 msgr2=0x7f3a4019bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:19.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.215+0000 7f3a4836c700 1 --2- 172.21.15.88:0/1670459941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a401007d0 0x7f3a4019bde0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3a3800b9a0 tx=0x7f3a3800bcb0 comp rx=0 tx=0).stop 2023-07-14T17:44:19.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.215+0000 7f3a4836c700 1 -- 172.21.15.88:0/1670459941 shutdown_connections 2023-07-14T17:44:19.198 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.215+0000 7f3a4836c700 1 --2- 172.21.15.88:0/1670459941 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3a2c0a11d0 0x7f3a2c0a3680 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:19.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.215+0000 7f3a4836c700 1 --2- 172.21.15.88:0/1670459941 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a401007d0 0x7f3a4019bde0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:19.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.215+0000 7f3a4836c700 1 --2- 172.21.15.88:0/1670459941 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a400fec40 0x7f3a40194820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:19.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.215+0000 7f3a4836c700 1 --2- 172.21.15.88:0/1670459941 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a400ffe40 0x7f3a40194d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:19.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.215+0000 7f3a4836c700 1 -- 172.21.15.88:0/1670459941 >> 172.21.15.88:0/1670459941 conn(0x7f3a400fa290 msgr2=0x7f3a400fc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:19.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.215+0000 7f3a4836c700 1 -- 172.21.15.88:0/1670459941 shutdown_connections 2023-07-14T17:44:19.199 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:19.215+0000 7f3a4836c700 1 -- 172.21.15.88:0/1670459941 wait complete. 2023-07-14T17:44:19.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:18 smithi178 ceph-mon[129368]: pgmap v305: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 511 B/s wr, 9 op/s 2023-07-14T17:44:19.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:18 smithi107 ceph-mon[128999]: pgmap v305: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 511 B/s wr, 9 op/s 2023-07-14T17:44:19.609 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-14T17:44:19.620 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi088.front.sepia.ceph.com 2023-07-14T17:44:19.621 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- bash -c 'ceph fs dump' 2023-07-14T17:44:20.174 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:44:20.384 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:20 smithi088 ceph-mon[127465]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:44:20.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:20 smithi178 ceph-mon[129368]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:44:20.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:20 smithi107 ceph-mon[128999]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:44:21.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.390+0000 7f0555fd8700 1 -- 172.21.15.88:0/3131158436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05500ffdb0 msgr2=0x7f0550100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:21.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.390+0000 7f0555fd8700 1 --2- 172.21.15.88:0/3131158436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05500ffdb0 0x7f0550100200 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f0538009990 tx=0x7f0538009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:21.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.390+0000 7f0555fd8700 1 -- 172.21.15.88:0/3131158436 shutdown_connections 2023-07-14T17:44:21.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.390+0000 7f0555fd8700 1 --2- 172.21.15.88:0/3131158436 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0550100740 0x7f0550107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:21.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.390+0000 7f0555fd8700 1 --2- 172.21.15.88:0/3131158436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05500febb0 0x7f05500fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:21.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.390+0000 7f0555fd8700 1 --2- 172.21.15.88:0/3131158436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f05500ffdb0 0x7f0550100200 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:21.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.390+0000 7f0555fd8700 1 -- 172.21.15.88:0/3131158436 >> 172.21.15.88:0/3131158436 conn(0x7f05500fa1e0 msgr2=0x7f05500fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:21.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.390+0000 7f0555fd8700 1 -- 172.21.15.88:0/3131158436 shutdown_connections 2023-07-14T17:44:21.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.390+0000 7f0555fd8700 1 -- 172.21.15.88:0/3131158436 wait complete. 2023-07-14T17:44:21.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.391+0000 7f0555fd8700 1 Processor -- start 2023-07-14T17:44:21.375 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.391+0000 7f0555fd8700 1 -- start start 2023-07-14T17:44:21.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f0555fd8700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05500febb0 0x7f0550194860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:21.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f0555fd8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05500ffdb0 0x7f0550194da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:21.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f0555fd8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0550100740 0x7f055019be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:21.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f0555fd8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055019c3f0 con 0x7f0550100740 2023-07-14T17:44:21.377 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f0555fd8700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055019c530 con 0x7f05500ffdb0 2023-07-14T17:44:21.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f0555fd8700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055019c6c0 con 0x7f05500febb0 2023-07-14T17:44:21.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f054ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0550100740 0x7f055019be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:21.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f054ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0550100740 0x7f055019be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42984/0 (socket says 172.21.15.88:42984) 2023-07-14T17:44:21.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f054ffff700 1 -- 172.21.15.88:0/3983700366 learned_addr learned my addr 172.21.15.88:0/3983700366 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:21.378 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f054effd700 1 --2- 172.21.15.88:0/3983700366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05500ffdb0 0x7f0550194da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:21.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f054f7fe700 1 --2- 172.21.15.88:0/3983700366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05500febb0 0x7f0550194860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:21.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f054ffff700 1 -- 172.21.15.88:0/3983700366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05500febb0 msgr2=0x7f0550194860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:21.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f054ffff700 1 --2- 172.21.15.88:0/3983700366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05500febb0 0x7f0550194860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:21.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f054ffff700 1 -- 172.21.15.88:0/3983700366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05500ffdb0 msgr2=0x7f0550194da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:21.379 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f054ffff700 1 --2- 172.21.15.88:0/3983700366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05500ffdb0 0x7f0550194da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:21.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f054ffff700 1 -- 172.21.15.88:0/3983700366 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05380095f0 con 0x7f0550100740 2023-07-14T17:44:21.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f054ffff700 1 --2- 172.21.15.88:0/3983700366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0550100740 0x7f055019be20 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f054800add0 tx=0x7f054800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:21.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f054cff9700 1 -- 172.21.15.88:0/3983700366 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054800f070 con 0x7f0550100740 2023-07-14T17:44:21.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f054cff9700 1 -- 172.21.15.88:0/3983700366 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0548007b30 con 0x7f0550100740 2023-07-14T17:44:21.380 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f0555fd8700 1 -- 172.21.15.88:0/3983700366 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05500689f0 con 0x7f0550100740 2023-07-14T17:44:21.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.392+0000 7f054cff9700 1 -- 172.21.15.88:0/3983700366 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0548005280 con 0x7f0550100740 2023-07-14T17:44:21.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.393+0000 7f0555fd8700 1 -- 172.21.15.88:0/3983700366 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0550068ef0 con 0x7f0550100740 2023-07-14T17:44:21.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.393+0000 7f0555fd8700 1 -- 172.21.15.88:0/3983700366 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f055010a270 con 0x7f0550100740 2023-07-14T17:44:21.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.394+0000 7f054cff9700 1 -- 172.21.15.88:0/3983700366 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f054800c890 con 0x7f0550100740 2023-07-14T17:44:21.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.395+0000 7f054cff9700 1 --2- 172.21.15.88:0/3983700366 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f053c0a1150 0x7f053c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:21.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.395+0000 7f054cff9700 1 -- 172.21.15.88:0/3983700366 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f05480cc6e0 con 0x7f0550100740 2023-07-14T17:44:21.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.395+0000 7f054f7fe700 1 --2- 172.21.15.88:0/3983700366 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f053c0a1150 0x7f053c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:21.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.395+0000 7f054f7fe700 1 --2- 172.21.15.88:0/3983700366 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f053c0a1150 0x7f053c0a3600 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f05500ffc10 tx=0x7f0540007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:21.383 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.398+0000 7f054cff9700 1 -- 172.21.15.88:0/3983700366 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0548095bb0 con 0x7f0550100740 2023-07-14T17:44:21.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:21 smithi178 ceph-mon[129368]: pgmap v306: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 596 B/s wr, 10 op/s 2023-07-14T17:44:21.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:21 smithi088 ceph-mon[127465]: pgmap v306: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 596 B/s wr, 10 op/s 2023-07-14T17:44:21.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:21 smithi107 ceph-mon[128999]: pgmap v306: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 596 B/s wr, 10 op/s 2023-07-14T17:44:21.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.627+0000 7f0555fd8700 1 -- 172.21.15.88:0/3983700366 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0550105bf0 con 0x7f0550100740 2023-07-14T17:44:21.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.627+0000 7f054cff9700 1 -- 172.21.15.88:0/3983700366 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+2932 (secure 0 0 0) 0x7f05480991d0 con 0x7f0550100740 2023-07-14T17:44:21.610 INFO:teuthology.orchestra.run.smithi088.stdout:e27 2023-07-14T17:44:21.610 INFO:teuthology.orchestra.run.smithi088.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T17:44:21.610 INFO:teuthology.orchestra.run.smithi088.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T17:44:21.610 INFO:teuthology.orchestra.run.smithi088.stdout:legacy client fscid: 1 2023-07-14T17:44:21.611 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:21.611 INFO:teuthology.orchestra.run.smithi088.stdout:Filesystem 'cephfs' (1) 2023-07-14T17:44:21.611 INFO:teuthology.orchestra.run.smithi088.stdout:fs_name cephfs 2023-07-14T17:44:21.611 INFO:teuthology.orchestra.run.smithi088.stdout:epoch 27 2023-07-14T17:44:21.611 INFO:teuthology.orchestra.run.smithi088.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-07-14T17:44:21.611 INFO:teuthology.orchestra.run.smithi088.stdout:created 2023-07-14T17:42:42.314442+0000 2023-07-14T17:44:21.611 INFO:teuthology.orchestra.run.smithi088.stdout:modified 2023-07-14T17:43:04.774381+0000 2023-07-14T17:44:21.612 INFO:teuthology.orchestra.run.smithi088.stdout:tableserver 0 2023-07-14T17:44:21.612 INFO:teuthology.orchestra.run.smithi088.stdout:root 0 2023-07-14T17:44:21.612 INFO:teuthology.orchestra.run.smithi088.stdout:session_timeout 300 2023-07-14T17:44:21.612 INFO:teuthology.orchestra.run.smithi088.stdout:session_autoclose 300 2023-07-14T17:44:21.612 INFO:teuthology.orchestra.run.smithi088.stdout:max_file_size 1099511627776 2023-07-14T17:44:21.612 INFO:teuthology.orchestra.run.smithi088.stdout:max_xattr_size 65536 2023-07-14T17:44:21.612 INFO:teuthology.orchestra.run.smithi088.stdout:required_client_features {} 2023-07-14T17:44:21.612 INFO:teuthology.orchestra.run.smithi088.stdout:last_failure 0 2023-07-14T17:44:21.613 INFO:teuthology.orchestra.run.smithi088.stdout:last_failure_osd_epoch 0 2023-07-14T17:44:21.613 INFO:teuthology.orchestra.run.smithi088.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T17:44:21.613 INFO:teuthology.orchestra.run.smithi088.stdout:max_mds 5 2023-07-14T17:44:21.613 INFO:teuthology.orchestra.run.smithi088.stdout:in 0,1,2,3,4 2023-07-14T17:44:21.613 INFO:teuthology.orchestra.run.smithi088.stdout:up {0=24506,1=24473,2=24469,3=24454,4=24449} 2023-07-14T17:44:21.613 INFO:teuthology.orchestra.run.smithi088.stdout:failed 2023-07-14T17:44:21.613 INFO:teuthology.orchestra.run.smithi088.stdout:damaged 2023-07-14T17:44:21.614 INFO:teuthology.orchestra.run.smithi088.stdout:stopped 2023-07-14T17:44:21.614 INFO:teuthology.orchestra.run.smithi088.stdout:data_pools [3,4] 2023-07-14T17:44:21.614 INFO:teuthology.orchestra.run.smithi088.stdout:metadata_pool 2 2023-07-14T17:44:21.614 INFO:teuthology.orchestra.run.smithi088.stdout:inline_data disabled 2023-07-14T17:44:21.614 INFO:teuthology.orchestra.run.smithi088.stdout:balancer 2023-07-14T17:44:21.614 INFO:teuthology.orchestra.run.smithi088.stdout:bal_rank_mask -1 2023-07-14T17:44:21.615 INFO:teuthology.orchestra.run.smithi088.stdout:standby_count_wanted 0 2023-07-14T17:44:21.615 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.b{0:24506} state up:active seq 3 addr [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T17:44:21.615 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.f{0:24442} state up:standby-replay seq 1 addr [v2:172.21.15.178:6834/2332025068,v1:172.21.15.178:6835/2332025068] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T17:44:21.615 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.h{1:24473} state up:active seq 10 addr [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T17:44:21.615 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.k{1:24419} state up:standby-replay seq 1 addr [v2:172.21.15.107:6832/2946270337,v1:172.21.15.107:6833/2946270337] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T17:44:21.615 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.i{2:24469} state up:active seq 9 addr [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T17:44:21.615 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.l{2:24415} state up:standby-replay seq 1 addr [v2:172.21.15.178:6832/1716372869,v1:172.21.15.178:6833/1716372869] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T17:44:21.615 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.c{3:24454} state up:active seq 11 addr [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T17:44:21.616 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.a{3:14628} state up:standby-replay seq 1 addr [v2:172.21.15.88:6840/2509502918,v1:172.21.15.88:6841/2509502918] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T17:44:21.616 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.e{4:24449} state up:active seq 12 addr [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T17:44:21.616 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.g{4:14616} state up:standby-replay seq 1 addr [v2:172.21.15.88:6838/3046592155,v1:172.21.15.88:6839/3046592155] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T17:44:21.616 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:21.616 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:21.616 INFO:teuthology.orchestra.run.smithi088.stdout:Standby daemons: 2023-07-14T17:44:21.616 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:21.617 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.j{-1:14544} state up:standby seq 1 addr [v2:172.21.15.88:6834/2870142745,v1:172.21.15.88:6835/2870142745] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T17:44:21.617 INFO:teuthology.orchestra.run.smithi088.stdout:[mds.d{-1:14550} state up:standby seq 1 addr [v2:172.21.15.88:6836/783713410,v1:172.21.15.88:6837/783713410] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T17:44:21.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.629+0000 7f0555fd8700 1 -- 172.21.15.88:0/3983700366 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f053c0a1150 msgr2=0x7f053c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:21.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.629+0000 7f0555fd8700 1 --2- 172.21.15.88:0/3983700366 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f053c0a1150 0x7f053c0a3600 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f05500ffc10 tx=0x7f0540007400 comp rx=0 tx=0).stop 2023-07-14T17:44:21.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.629+0000 7f0555fd8700 1 -- 172.21.15.88:0/3983700366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0550100740 msgr2=0x7f055019be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:21.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.629+0000 7f0555fd8700 1 --2- 172.21.15.88:0/3983700366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0550100740 0x7f055019be20 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f054800add0 tx=0x7f054800c330 comp rx=0 tx=0).stop 2023-07-14T17:44:21.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.629+0000 7f0555fd8700 1 -- 172.21.15.88:0/3983700366 shutdown_connections 2023-07-14T17:44:21.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.629+0000 7f0555fd8700 1 --2- 172.21.15.88:0/3983700366 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f053c0a1150 0x7f053c0a3600 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:21.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.629+0000 7f0555fd8700 1 --2- 172.21.15.88:0/3983700366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0550100740 0x7f055019be20 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:21.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.629+0000 7f0555fd8700 1 --2- 172.21.15.88:0/3983700366 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f05500febb0 0x7f0550194860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:21.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.629+0000 7f0555fd8700 1 --2- 172.21.15.88:0/3983700366 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f05500ffdb0 0x7f0550194da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:21.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.629+0000 7f0555fd8700 1 -- 172.21.15.88:0/3983700366 >> 172.21.15.88:0/3983700366 conn(0x7f05500fa1e0 msgr2=0x7f05500fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:21.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.629+0000 7f0555fd8700 1 -- 172.21.15.88:0/3983700366 shutdown_connections 2023-07-14T17:44:21.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:21.629+0000 7f0555fd8700 1 -- 172.21.15.88:0/3983700366 wait complete. 2023-07-14T17:44:21.619 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 27 2023-07-14T17:44:22.024 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b462a556-226c-11ee-9b33-001a4aab830c -- bash -c 'ceph osd dump' 2023-07-14T17:44:22.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:22 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3983700366' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T17:44:22.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:22 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3983700366' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T17:44:22.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:22 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3983700366' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T17:44:22.576 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:44:23.069 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:23 smithi088 ceph-mon[127465]: pgmap v307: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 596 B/s wr, 12 op/s 2023-07-14T17:44:23.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:23 smithi178 ceph-mon[129368]: pgmap v307: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 596 B/s wr, 12 op/s 2023-07-14T17:44:23.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:23 smithi107 ceph-mon[128999]: pgmap v307: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 596 B/s wr, 12 op/s 2023-07-14T17:44:23.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.810+0000 7f195bd50700 1 -- 172.21.15.88:0/2374124744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19540ff7c0 msgr2=0x7f1954107070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:23.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.810+0000 7f195bd50700 1 --2- 172.21.15.88:0/2374124744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19540ff7c0 0x7f1954107070 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f194c00b0a0 tx=0x7f194c00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:23.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.810+0000 7f195bd50700 1 -- 172.21.15.88:0/2374124744 shutdown_connections 2023-07-14T17:44:23.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.810+0000 7f195bd50700 1 --2- 172.21.15.88:0/2374124744 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19540ff7c0 0x7f1954107070 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:23.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.810+0000 7f195bd50700 1 --2- 172.21.15.88:0/2374124744 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19541001b0 0x7f19541005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:23.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.810+0000 7f195bd50700 1 --2- 172.21.15.88:0/2374124744 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19540fee10 0x7f19540ff280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:23.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.810+0000 7f195bd50700 1 -- 172.21.15.88:0/2374124744 >> 172.21.15.88:0/2374124744 conn(0x7f19540f99b0 msgr2=0x7f19540fbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:23.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.810+0000 7f195bd50700 1 -- 172.21.15.88:0/2374124744 shutdown_connections 2023-07-14T17:44:23.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.811+0000 7f195bd50700 1 -- 172.21.15.88:0/2374124744 wait complete. 2023-07-14T17:44:23.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.811+0000 7f195bd50700 1 Processor -- start 2023-07-14T17:44:23.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.811+0000 7f195bd50700 1 -- start start 2023-07-14T17:44:23.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.811+0000 7f195bd50700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19540fee10 0x7f19540fe8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:23.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.811+0000 7f195bd50700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19540ff7c0 0x7f195419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:23.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.811+0000 7f195bd50700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19541001b0 0x7f195419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:23.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.811+0000 7f195bd50700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195419d500 con 0x7f19540ff7c0 2023-07-14T17:44:23.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.811+0000 7f195bd50700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195419d670 con 0x7f19541001b0 2023-07-14T17:44:23.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.811+0000 7f195bd50700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195419d830 con 0x7f19540fee10 2023-07-14T17:44:23.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.811+0000 7f19592eb700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19540ff7c0 0x7f195419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:23.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.811+0000 7f19592eb700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19540ff7c0 0x7f195419a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:42994/0 (socket says 172.21.15.88:42994) 2023-07-14T17:44:23.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.811+0000 7f19592eb700 1 -- 172.21.15.88:0/1690587072 learned_addr learned my addr 172.21.15.88:0/1690587072 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:23.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.812+0000 7f195a2ed700 1 --2- 172.21.15.88:0/1690587072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19541001b0 0x7f195419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:23.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.812+0000 7f1959aec700 1 --2- 172.21.15.88:0/1690587072 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19540fee10 0x7f19540fe8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:23.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.812+0000 7f19592eb700 1 -- 172.21.15.88:0/1690587072 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19540fee10 msgr2=0x7f19540fe8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:23.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.812+0000 7f19592eb700 1 --2- 172.21.15.88:0/1690587072 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19540fee10 0x7f19540fe8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:23.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.812+0000 7f19592eb700 1 -- 172.21.15.88:0/1690587072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19541001b0 msgr2=0x7f195419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:23.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.812+0000 7f19592eb700 1 --2- 172.21.15.88:0/1690587072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19541001b0 0x7f195419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:23.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.812+0000 7f19592eb700 1 -- 172.21.15.88:0/1690587072 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194c009d00 con 0x7f19540ff7c0 2023-07-14T17:44:23.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.812+0000 7f19592eb700 1 --2- 172.21.15.88:0/1690587072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19540ff7c0 0x7f195419a6d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f195000d950 tx=0x7f195000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:23.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.812+0000 7f194affd700 1 -- 172.21.15.88:0/1690587072 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1950009b40 con 0x7f19540ff7c0 2023-07-14T17:44:23.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.812+0000 7f194affd700 1 -- 172.21.15.88:0/1690587072 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1950009ca0 con 0x7f19540ff7c0 2023-07-14T17:44:23.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.812+0000 7f194affd700 1 -- 172.21.15.88:0/1690587072 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195000f620 con 0x7f19540ff7c0 2023-07-14T17:44:23.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.812+0000 7f195bd50700 1 -- 172.21.15.88:0/1690587072 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f195419db10 con 0x7f19540ff7c0 2023-07-14T17:44:23.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.812+0000 7f195bd50700 1 -- 172.21.15.88:0/1690587072 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195419dfc0 con 0x7f19540ff7c0 2023-07-14T17:44:23.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.813+0000 7f195bd50700 1 -- 172.21.15.88:0/1690587072 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f195410a280 con 0x7f19540ff7c0 2023-07-14T17:44:23.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.814+0000 7f194affd700 1 -- 172.21.15.88:0/1690587072 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1950010460 con 0x7f19540ff7c0 2023-07-14T17:44:23.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.814+0000 7f194affd700 1 --2- 172.21.15.88:0/1690587072 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f19400a11d0 0x7f19400a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:23.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.815+0000 7f1959aec700 1 --2- 172.21.15.88:0/1690587072 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f19400a11d0 0x7f19400a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:23.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.815+0000 7f194affd700 1 -- 172.21.15.88:0/1690587072 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f19500ccfc0 con 0x7f19540ff7c0 2023-07-14T17:44:23.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.815+0000 7f1959aec700 1 --2- 172.21.15.88:0/1690587072 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f19400a11d0 0x7f19400a3680 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1944005950 tx=0x7f19440058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:23.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:23.818+0000 7f194affd700 1 -- 172.21.15.88:0/1690587072 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f19500963a0 con 0x7f19540ff7c0 2023-07-14T17:44:23.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.006+0000 7f195bd50700 1 -- 172.21.15.88:0/1690587072 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f195410a4a0 con 0x7f19540ff7c0 2023-07-14T17:44:23.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.007+0000 7f194affd700 1 -- 172.21.15.88:0/1690587072 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v88) v1 ==== 56+0+5610 (secure 0 0 0) 0x7f1950020070 con 0x7f19540ff7c0 2023-07-14T17:44:23.989 INFO:teuthology.orchestra.run.smithi088.stdout:epoch 88 2023-07-14T17:44:23.989 INFO:teuthology.orchestra.run.smithi088.stdout:fsid b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:44:23.989 INFO:teuthology.orchestra.run.smithi088.stdout:created 2023-07-14T17:35:48.898032+0000 2023-07-14T17:44:23.990 INFO:teuthology.orchestra.run.smithi088.stdout:modified 2023-07-14T17:43:44.887959+0000 2023-07-14T17:44:23.990 INFO:teuthology.orchestra.run.smithi088.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-07-14T17:44:23.990 INFO:teuthology.orchestra.run.smithi088.stdout:crush_version 27 2023-07-14T17:44:23.990 INFO:teuthology.orchestra.run.smithi088.stdout:full_ratio 0.95 2023-07-14T17:44:23.990 INFO:teuthology.orchestra.run.smithi088.stdout:backfillfull_ratio 0.9 2023-07-14T17:44:23.990 INFO:teuthology.orchestra.run.smithi088.stdout:nearfull_ratio 0.85 2023-07-14T17:44:23.990 INFO:teuthology.orchestra.run.smithi088.stdout:require_min_compat_client luminous 2023-07-14T17:44:23.991 INFO:teuthology.orchestra.run.smithi088.stdout:min_compat_client luminous 2023-07-14T17:44:23.991 INFO:teuthology.orchestra.run.smithi088.stdout:require_osd_release reef 2023-07-14T17:44:23.991 INFO:teuthology.orchestra.run.smithi088.stdout:stretch_mode_enabled false 2023-07-14T17:44:23.991 INFO:teuthology.orchestra.run.smithi088.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2023-07-14T17:44:23.991 INFO:teuthology.orchestra.run.smithi088.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 81 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2023-07-14T17:44:23.991 INFO:teuthology.orchestra.run.smithi088.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2023-07-14T17:44:23.991 INFO:teuthology.orchestra.run.smithi088.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 86 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2023-07-14T17:44:23.991 INFO:teuthology.orchestra.run.smithi088.stdout:max_osd 12 2023-07-14T17:44:23.992 INFO:teuthology.orchestra.run.smithi088.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.88:6802/826983341,v1:172.21.15.88:6803/826983341] [v2:172.21.15.88:6804/826983341,v1:172.21.15.88:6805/826983341] exists,up 32bbe236-4df9-42d6-8541-5d24d944eeee 2023-07-14T17:44:23.992 INFO:teuthology.orchestra.run.smithi088.stdout:osd.1 up in weight 1 up_from 13 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6800/1795455333,v1:172.21.15.107:6801/1795455333] [v2:172.21.15.107:6802/1795455333,v1:172.21.15.107:6803/1795455333] exists,up 7d11240d-b997-40ba-b67b-f39d92dce9b9 2023-07-14T17:44:23.992 INFO:teuthology.orchestra.run.smithi088.stdout:osd.2 up in weight 1 up_from 18 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.178:6800/4021325462,v1:172.21.15.178:6801/4021325462] [v2:172.21.15.178:6802/4021325462,v1:172.21.15.178:6803/4021325462] exists,up 41542091-8564-496c-9b43-a827f60ac844 2023-07-14T17:44:23.992 INFO:teuthology.orchestra.run.smithi088.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.88:6810/3836966618,v1:172.21.15.88:6811/3836966618] [v2:172.21.15.88:6812/3836966618,v1:172.21.15.88:6813/3836966618] exists,up 77c095e1-8805-435b-bb0e-1ff3ef4ba0a2 2023-07-14T17:44:23.992 INFO:teuthology.orchestra.run.smithi088.stdout:osd.4 up in weight 1 up_from 32 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6808/2380989846,v1:172.21.15.107:6809/2380989846] [v2:172.21.15.107:6810/2380989846,v1:172.21.15.107:6811/2380989846] exists,up ae0f806e-a7ef-445d-a7d4-14823157b841 2023-07-14T17:44:23.992 INFO:teuthology.orchestra.run.smithi088.stdout:osd.5 up in weight 1 up_from 37 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.178:6808/4006439725,v1:172.21.15.178:6809/4006439725] [v2:172.21.15.178:6810/4006439725,v1:172.21.15.178:6811/4006439725] exists,up b926d4fa-a9de-4bb6-8e00-30a2b8a35b36 2023-07-14T17:44:23.992 INFO:teuthology.orchestra.run.smithi088.stdout:osd.6 up in weight 1 up_from 44 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.88:6818/1424514128,v1:172.21.15.88:6819/1424514128] [v2:172.21.15.88:6820/1424514128,v1:172.21.15.88:6821/1424514128] exists,up 206cd369-2456-41c5-8a3f-d1efce5bb45a 2023-07-14T17:44:23.993 INFO:teuthology.orchestra.run.smithi088.stdout:osd.7 up in weight 1 up_from 49 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6816/2214437741,v1:172.21.15.107:6817/2214437741] [v2:172.21.15.107:6818/2214437741,v1:172.21.15.107:6819/2214437741] exists,up 59b9919f-52fb-42b6-b1d4-dee1df75c23a 2023-07-14T17:44:23.993 INFO:teuthology.orchestra.run.smithi088.stdout:osd.8 up in weight 1 up_from 57 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.178:6816/2305567673,v1:172.21.15.178:6817/2305567673] [v2:172.21.15.178:6818/2305567673,v1:172.21.15.178:6819/2305567673] exists,up 81ec848e-9acc-40e8-8890-3d80f6844597 2023-07-14T17:44:23.993 INFO:teuthology.orchestra.run.smithi088.stdout:osd.9 up in weight 1 up_from 64 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.88:6826/3211834989,v1:172.21.15.88:6827/3211834989] [v2:172.21.15.88:6828/3211834989,v1:172.21.15.88:6829/3211834989] exists,up 96a3d388-7f40-40ba-80ac-b670a13a9993 2023-07-14T17:44:23.993 INFO:teuthology.orchestra.run.smithi088.stdout:osd.10 up in weight 1 up_from 69 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6824/1223599278,v1:172.21.15.107:6825/1223599278] [v2:172.21.15.107:6826/1223599278,v1:172.21.15.107:6827/1223599278] exists,up bc27ef27-7791-4c70-be0b-84e60b24af36 2023-07-14T17:44:23.993 INFO:teuthology.orchestra.run.smithi088.stdout:osd.11 up in weight 1 up_from 77 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.178:6824/1862417235,v1:172.21.15.178:6825/1862417235] [v2:172.21.15.178:6826/1862417235,v1:172.21.15.178:6827/1862417235] exists,up ba0e783d-7fbe-420c-93d9-81923f17bb9f 2023-07-14T17:44:23.993 INFO:teuthology.orchestra.run.smithi088.stdout:pg_upmap_items 2.a [2,11] 2023-07-14T17:44:23.993 INFO:teuthology.orchestra.run.smithi088.stdout:pg_upmap_items 2.16 [2,4] 2023-07-14T17:44:23.994 INFO:teuthology.orchestra.run.smithi088.stdout:pg_upmap_items 2.1d [8,11] 2023-07-14T17:44:23.994 INFO:teuthology.orchestra.run.smithi088.stdout:pg_upmap_items 3.3 [4,9] 2023-07-14T17:44:23.994 INFO:teuthology.orchestra.run.smithi088.stdout:pg_upmap_items 3.6 [4,6] 2023-07-14T17:44:23.994 INFO:teuthology.orchestra.run.smithi088.stdout:pg_upmap_items 3.1e [5,8] 2023-07-14T17:44:23.994 INFO:teuthology.orchestra.run.smithi088.stdout:pg_upmap_items 3.2c [4,9] 2023-07-14T17:44:23.994 INFO:teuthology.orchestra.run.smithi088.stdout:pg_upmap_items 3.38 [3,9] 2023-07-14T17:44:23.994 INFO:teuthology.orchestra.run.smithi088.stdout:pg_upmap_items 4.6 [9,11] 2023-07-14T17:44:23.994 INFO:teuthology.orchestra.run.smithi088.stdout:pg_upmap_items 4.14 [9,8] 2023-07-14T17:44:23.995 INFO:teuthology.orchestra.run.smithi088.stdout:blocklist 172.21.15.88:0/1441439704 expires 2023-07-15T17:36:43.799741+0000 2023-07-14T17:44:23.995 INFO:teuthology.orchestra.run.smithi088.stdout:blocklist 172.21.15.88:0/3897985004 expires 2023-07-15T17:36:43.799741+0000 2023-07-14T17:44:23.995 INFO:teuthology.orchestra.run.smithi088.stdout:blocklist 172.21.15.88:6801/2424184187 expires 2023-07-15T17:36:43.799741+0000 2023-07-14T17:44:23.995 INFO:teuthology.orchestra.run.smithi088.stdout:blocklist 172.21.15.88:6801/1369181448 expires 2023-07-15T17:36:21.153428+0000 2023-07-14T17:44:23.995 INFO:teuthology.orchestra.run.smithi088.stdout:blocklist 172.21.15.88:6800/1369181448 expires 2023-07-15T17:36:21.153428+0000 2023-07-14T17:44:23.995 INFO:teuthology.orchestra.run.smithi088.stdout:blocklist 172.21.15.88:0/1790623516 expires 2023-07-15T17:36:21.153428+0000 2023-07-14T17:44:23.995 INFO:teuthology.orchestra.run.smithi088.stdout:blocklist 172.21.15.88:0/3232411191 expires 2023-07-15T17:36:21.153428+0000 2023-07-14T17:44:23.995 INFO:teuthology.orchestra.run.smithi088.stdout:blocklist 172.21.15.88:0/3329866608 expires 2023-07-15T17:36:21.153428+0000 2023-07-14T17:44:23.996 INFO:teuthology.orchestra.run.smithi088.stdout:blocklist 172.21.15.88:0/954461155 expires 2023-07-15T17:36:43.799741+0000 2023-07-14T17:44:23.996 INFO:teuthology.orchestra.run.smithi088.stdout:blocklist 172.21.15.88:6800/2424184187 expires 2023-07-15T17:36:43.799741+0000 2023-07-14T17:44:23.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.009+0000 7f195bd50700 1 -- 172.21.15.88:0/1690587072 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f19400a11d0 msgr2=0x7f19400a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:23.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.009+0000 7f195bd50700 1 --2- 172.21.15.88:0/1690587072 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f19400a11d0 0x7f19400a3680 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1944005950 tx=0x7f19440058e0 comp rx=0 tx=0).stop 2023-07-14T17:44:23.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.009+0000 7f195bd50700 1 -- 172.21.15.88:0/1690587072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19540ff7c0 msgr2=0x7f195419a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:23.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.009+0000 7f195bd50700 1 --2- 172.21.15.88:0/1690587072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19540ff7c0 0x7f195419a6d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f195000d950 tx=0x7f195000dd10 comp rx=0 tx=0).stop 2023-07-14T17:44:23.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.009+0000 7f195bd50700 1 -- 172.21.15.88:0/1690587072 shutdown_connections 2023-07-14T17:44:23.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.009+0000 7f195bd50700 1 --2- 172.21.15.88:0/1690587072 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f19541001b0 0x7f195419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:23.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.009+0000 7f195bd50700 1 --2- 172.21.15.88:0/1690587072 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f19540fee10 0x7f19540fe8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:23.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.009+0000 7f195bd50700 1 --2- 172.21.15.88:0/1690587072 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f19400a11d0 0x7f19400a3680 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:23.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.009+0000 7f195bd50700 1 --2- 172.21.15.88:0/1690587072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f19540ff7c0 0x7f195419a6d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:23.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.009+0000 7f195bd50700 1 -- 172.21.15.88:0/1690587072 >> 172.21.15.88:0/1690587072 conn(0x7f19540f99b0 msgr2=0x7f19540fbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:23.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.009+0000 7f195bd50700 1 -- 172.21.15.88:0/1690587072 shutdown_connections 2023-07-14T17:44:23.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.009+0000 7f195bd50700 1 -- 172.21.15.88:0/1690587072 wait complete. 2023-07-14T17:44:24.321 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:24 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1690587072' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T17:44:24.323 INFO:teuthology.run_tasks:Running task fs.ready... 2023-07-14T17:44:24.334 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:44:24.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:24 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1690587072' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T17:44:24.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.478+0000 7fcc5daa0700 1 -- 172.21.15.88:0/3387098124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc580ffd30 msgr2=0x7fcc58100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:24.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.478+0000 7fcc5daa0700 1 --2- 172.21.15.88:0/3387098124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc580ffd30 0x7fcc58100180 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fcc48009990 tx=0x7fcc48009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:24.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.478+0000 7fcc5daa0700 1 -- 172.21.15.88:0/3387098124 shutdown_connections 2023-07-14T17:44:24.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.478+0000 7fcc5daa0700 1 --2- 172.21.15.88:0/3387098124 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc581006c0 0x7fcc58106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:24.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.478+0000 7fcc5daa0700 1 --2- 172.21.15.88:0/3387098124 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc580feb40 0x7fcc580fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:24.461 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.478+0000 7fcc5daa0700 1 --2- 172.21.15.88:0/3387098124 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc580ffd30 0x7fcc58100180 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:24.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.478+0000 7fcc5daa0700 1 -- 172.21.15.88:0/3387098124 >> 172.21.15.88:0/3387098124 conn(0x7fcc580fa160 msgr2=0x7fcc580fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:24.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.478+0000 7fcc5daa0700 1 -- 172.21.15.88:0/3387098124 shutdown_connections 2023-07-14T17:44:24.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.478+0000 7fcc5daa0700 1 -- 172.21.15.88:0/3387098124 wait complete. 2023-07-14T17:44:24.462 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.478+0000 7fcc5daa0700 1 Processor -- start 2023-07-14T17:44:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.478+0000 7fcc5daa0700 1 -- start start 2023-07-14T17:44:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc5daa0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc580feb40 0x7fcc580692e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc5daa0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc580ffd30 0x7fcc5819c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc5daa0700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc581006c0 0x7fcc5819eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc5daa0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5819f590 con 0x7fcc580feb40 2023-07-14T17:44:24.464 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc5daa0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5819f700 con 0x7fcc580ffd30 2023-07-14T17:44:24.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc5daa0700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5819f8c0 con 0x7fcc581006c0 2023-07-14T17:44:24.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc577fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc580feb40 0x7fcc580692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:24.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc577fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc580feb40 0x7fcc580692e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43004/0 (socket says 172.21.15.88:43004) 2023-07-14T17:44:24.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc577fe700 1 -- 172.21.15.88:0/1530593414 learned_addr learned my addr 172.21.15.88:0/1530593414 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:24.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc56ffd700 1 --2- 172.21.15.88:0/1530593414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc580ffd30 0x7fcc5819c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:24.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc57fff700 1 --2- 172.21.15.88:0/1530593414 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc581006c0 0x7fcc5819eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:24.465 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc577fe700 1 -- 172.21.15.88:0/1530593414 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc581006c0 msgr2=0x7fcc5819eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:24.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc577fe700 1 --2- 172.21.15.88:0/1530593414 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc581006c0 0x7fcc5819eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:24.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc577fe700 1 -- 172.21.15.88:0/1530593414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc580ffd30 msgr2=0x7fcc5819c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:24.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc577fe700 1 --2- 172.21.15.88:0/1530593414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc580ffd30 0x7fcc5819c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:24.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc577fe700 1 -- 172.21.15.88:0/1530593414 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc480095f0 con 0x7fcc580feb40 2023-07-14T17:44:24.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc57fff700 1 --2- 172.21.15.88:0/1530593414 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc581006c0 0x7fcc5819eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:24.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc577fe700 1 --2- 172.21.15.88:0/1530593414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc580feb40 0x7fcc580692e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00d980 tx=0x7fcc4c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:24.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc54ff9700 1 -- 172.21.15.88:0/1530593414 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c00de60 con 0x7fcc580feb40 2023-07-14T17:44:24.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc54ff9700 1 -- 172.21.15.88:0/1530593414 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcc4c0044b0 con 0x7fcc580feb40 2023-07-14T17:44:24.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc5daa0700 1 -- 172.21.15.88:0/1530593414 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc5819fba0 con 0x7fcc580feb40 2023-07-14T17:44:24.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc54ff9700 1 -- 172.21.15.88:0/1530593414 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c003b40 con 0x7fcc580feb40 2023-07-14T17:44:24.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.479+0000 7fcc5daa0700 1 -- 172.21.15.88:0/1530593414 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc581a00b0 con 0x7fcc580feb40 2023-07-14T17:44:24.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.480+0000 7fcc5daa0700 1 -- 172.21.15.88:0/1530593414 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5810a210 con 0x7fcc580feb40 2023-07-14T17:44:24.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.481+0000 7fcc54ff9700 1 -- 172.21.15.88:0/1530593414 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcc4c004870 con 0x7fcc580feb40 2023-07-14T17:44:24.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.482+0000 7fcc54ff9700 1 --2- 172.21.15.88:0/1530593414 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcc400a11a0 0x7fcc400a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:24.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.482+0000 7fcc56ffd700 1 --2- 172.21.15.88:0/1530593414 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcc400a11a0 0x7fcc400a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:24.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.482+0000 7fcc54ff9700 1 -- 172.21.15.88:0/1530593414 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcc4c0ccc40 con 0x7fcc580feb40 2023-07-14T17:44:24.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.482+0000 7fcc56ffd700 1 --2- 172.21.15.88:0/1530593414 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcc400a11a0 0x7fcc400a3650 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800b5c0 tx=0x7fcc48005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:24.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.485+0000 7fcc54ff9700 1 -- 172.21.15.88:0/1530593414 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcc4c096080 con 0x7fcc580feb40 2023-07-14T17:44:24.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:24 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1690587072' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T17:44:24.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.674+0000 7fcc5daa0700 1 -- 172.21.15.88:0/1530593414 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc5810a400 con 0x7fcc580feb40 2023-07-14T17:44:24.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.676+0000 7fcc54ff9700 1 -- 172.21.15.88:0/1530593414 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24512 (secure 0 0 0) 0x7fcc4c009940 con 0x7fcc580feb40 2023-07-14T17:44:24.659 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:24.659 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":88,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:43:44.887959+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:44:24.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.678+0000 7fcc5daa0700 1 -- 172.21.15.88:0/1530593414 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcc400a11a0 msgr2=0x7fcc400a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:24.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.678+0000 7fcc5daa0700 1 --2- 172.21.15.88:0/1530593414 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcc400a11a0 0x7fcc400a3650 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800b5c0 tx=0x7fcc48005fb0 comp rx=0 tx=0).stop 2023-07-14T17:44:24.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.678+0000 7fcc5daa0700 1 -- 172.21.15.88:0/1530593414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc580feb40 msgr2=0x7fcc580692e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:24.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.678+0000 7fcc5daa0700 1 --2- 172.21.15.88:0/1530593414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc580feb40 0x7fcc580692e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00d980 tx=0x7fcc4c00dd40 comp rx=0 tx=0).stop 2023-07-14T17:44:24.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.678+0000 7fcc5daa0700 1 -- 172.21.15.88:0/1530593414 shutdown_connections 2023-07-14T17:44:24.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.678+0000 7fcc5daa0700 1 --2- 172.21.15.88:0/1530593414 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fcc581006c0 0x7fcc5819eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:24.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.678+0000 7fcc5daa0700 1 --2- 172.21.15.88:0/1530593414 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fcc400a11a0 0x7fcc400a3650 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:24.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.678+0000 7fcc5daa0700 1 --2- 172.21.15.88:0/1530593414 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcc580feb40 0x7fcc580692e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:24.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.678+0000 7fcc5daa0700 1 --2- 172.21.15.88:0/1530593414 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fcc580ffd30 0x7fcc5819c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:24.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.678+0000 7fcc5daa0700 1 -- 172.21.15.88:0/1530593414 >> 172.21.15.88:0/1530593414 conn(0x7fcc580fa160 msgr2=0x7fcc580fc360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:24.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.679+0000 7fcc5daa0700 1 -- 172.21.15.88:0/1530593414 shutdown_connections 2023-07-14T17:44:24.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.679+0000 7fcc5daa0700 1 -- 172.21.15.88:0/1530593414 wait complete. 2023-07-14T17:44:24.674 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T17:38:50.126535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T17:42:40.725870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T17:42:41.512749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T17:42:44.764573+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T17:44:24.675 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T17:44:24.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.817+0000 7f5d92858700 1 -- 172.21.15.88:0/2655684730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d8c100100 msgr2=0x7f5d8c100510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:24.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.817+0000 7f5d92858700 1 --2- 172.21.15.88:0/2655684730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d8c100100 0x7f5d8c100510 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c009990 tx=0x7f5d7c009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:24.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.817+0000 7f5d92858700 1 -- 172.21.15.88:0/2655684730 shutdown_connections 2023-07-14T17:44:24.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.817+0000 7f5d92858700 1 --2- 172.21.15.88:0/2655684730 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5d8c0ff4d0 0x7f5d8c1070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:24.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.817+0000 7f5d92858700 1 --2- 172.21.15.88:0/2655684730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d8c100100 0x7f5d8c100510 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:24.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.817+0000 7f5d92858700 1 --2- 172.21.15.88:0/2655684730 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d8c100a50 0x7f5d8c0fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:24.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.817+0000 7f5d92858700 1 -- 172.21.15.88:0/2655684730 >> 172.21.15.88:0/2655684730 conn(0x7f5d8c0f98a0 msgr2=0x7f5d8c0fbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:24.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.817+0000 7f5d92858700 1 -- 172.21.15.88:0/2655684730 shutdown_connections 2023-07-14T17:44:24.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.817+0000 7f5d92858700 1 -- 172.21.15.88:0/2655684730 wait complete. 2023-07-14T17:44:24.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d92858700 1 Processor -- start 2023-07-14T17:44:24.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d92858700 1 -- start start 2023-07-14T17:44:24.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d92858700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d8c0ff4d0 0x7f5d8c195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:24.803 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d92858700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d8c100100 0x7f5d8c195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:24.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d92858700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5d8c100a50 0x7f5d8c19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:24.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d92858700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c19d480 con 0x7f5d8c100100 2023-07-14T17:44:24.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d92858700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c19d5c0 con 0x7f5d8c0ff4d0 2023-07-14T17:44:24.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d92858700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c19d750 con 0x7f5d8c100a50 2023-07-14T17:44:24.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d8b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d8c100100 0x7f5d8c195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:24.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d8b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d8c100100 0x7f5d8c195e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43018/0 (socket says 172.21.15.88:43018) 2023-07-14T17:44:24.804 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d8b7fe700 1 -- 172.21.15.88:0/1130538240 learned_addr learned my addr 172.21.15.88:0/1130538240 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:24.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d90df5700 1 --2- 172.21.15.88:0/1130538240 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5d8c100a50 0x7f5d8c19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:24.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d8bfff700 1 --2- 172.21.15.88:0/1130538240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d8c0ff4d0 0x7f5d8c195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:24.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d8b7fe700 1 -- 172.21.15.88:0/1130538240 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5d8c100a50 msgr2=0x7f5d8c19ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:24.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d8b7fe700 1 --2- 172.21.15.88:0/1130538240 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5d8c100a50 0x7f5d8c19ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:24.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d8b7fe700 1 -- 172.21.15.88:0/1130538240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d8c0ff4d0 msgr2=0x7f5d8c195900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:24.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d8b7fe700 1 --2- 172.21.15.88:0/1130538240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d8c0ff4d0 0x7f5d8c195900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:24.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d8b7fe700 1 -- 172.21.15.88:0/1130538240 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d80009520 con 0x7f5d8c100100 2023-07-14T17:44:24.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.818+0000 7f5d8bfff700 1 --2- 172.21.15.88:0/1130538240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d8c0ff4d0 0x7f5d8c195900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:24.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.819+0000 7f5d8b7fe700 1 --2- 172.21.15.88:0/1130538240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d8c100100 0x7f5d8c195e40 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5d8000ea30 tx=0x7f5d8000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:24.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.819+0000 7f5d897fa700 1 -- 172.21.15.88:0/1130538240 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80009820 con 0x7f5d8c100100 2023-07-14T17:44:24.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.819+0000 7f5d897fa700 1 -- 172.21.15.88:0/1130538240 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5d800044b0 con 0x7f5d8c100100 2023-07-14T17:44:24.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.819+0000 7f5d92858700 1 -- 172.21.15.88:0/1130538240 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7c0095f0 con 0x7f5d8c100100 2023-07-14T17:44:24.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.819+0000 7f5d897fa700 1 -- 172.21.15.88:0/1130538240 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80003ce0 con 0x7f5d8c100100 2023-07-14T17:44:24.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.819+0000 7f5d92858700 1 -- 172.21.15.88:0/1130538240 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8c19de00 con 0x7f5d8c100100 2023-07-14T17:44:24.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.820+0000 7f5d92858700 1 -- 172.21.15.88:0/1130538240 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8c1061e0 con 0x7f5d8c100100 2023-07-14T17:44:24.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.821+0000 7f5d897fa700 1 -- 172.21.15.88:0/1130538240 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5d80003e60 con 0x7f5d8c100100 2023-07-14T17:44:24.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.821+0000 7f5d897fa700 1 --2- 172.21.15.88:0/1130538240 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5d740a11d0 0x7f5d740a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:24.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.821+0000 7f5d8bfff700 1 --2- 172.21.15.88:0/1130538240 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5d740a11d0 0x7f5d740a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:24.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.821+0000 7f5d897fa700 1 -- 172.21.15.88:0/1130538240 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5d80014070 con 0x7f5d8c100100 2023-07-14T17:44:24.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.822+0000 7f5d8bfff700 1 --2- 172.21.15.88:0/1130538240 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5d740a11d0 0x7f5d740a3680 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c00b2e0 tx=0x7f5d7c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:24.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:24.824+0000 7f5d897fa700 1 -- 172.21.15.88:0/1130538240 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5d80096440 con 0x7f5d8c100100 2023-07-14T17:44:24.998 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.016+0000 7f5d92858700 1 -- 172.21.15.88:0/1130538240 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5d8c02cd70 con 0x7f5d8c100100 2023-07-14T17:44:25.003 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.021+0000 7f5d897fa700 1 -- 172.21.15.88:0/1130538240 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5d80099a60 con 0x7f5d8c100100 2023-07-14T17:44:25.003 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-07-14T17:44:25.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.023+0000 7f5d92858700 1 -- 172.21.15.88:0/1130538240 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5d740a11d0 msgr2=0x7f5d740a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:25.005 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.023+0000 7f5d92858700 1 --2- 172.21.15.88:0/1130538240 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5d740a11d0 0x7f5d740a3680 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c00b2e0 tx=0x7f5d7c005fd0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.023+0000 7f5d92858700 1 -- 172.21.15.88:0/1130538240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d8c100100 msgr2=0x7f5d8c195e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.023+0000 7f5d92858700 1 --2- 172.21.15.88:0/1130538240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d8c100100 0x7f5d8c195e40 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5d8000ea30 tx=0x7f5d8000ed40 comp rx=0 tx=0).stop 2023-07-14T17:44:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.023+0000 7f5d92858700 1 -- 172.21.15.88:0/1130538240 shutdown_connections 2023-07-14T17:44:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.023+0000 7f5d92858700 1 --2- 172.21.15.88:0/1130538240 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5d8c100a50 0x7f5d8c19ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.023+0000 7f5d92858700 1 --2- 172.21.15.88:0/1130538240 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5d8c0ff4d0 0x7f5d8c195900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.023+0000 7f5d92858700 1 --2- 172.21.15.88:0/1130538240 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f5d740a11d0 0x7f5d740a3680 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.006 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.023+0000 7f5d92858700 1 --2- 172.21.15.88:0/1130538240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5d8c100100 0x7f5d8c195e40 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.023+0000 7f5d92858700 1 -- 172.21.15.88:0/1130538240 >> 172.21.15.88:0/1130538240 conn(0x7f5d8c0f98a0 msgr2=0x7f5d8c103320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.023+0000 7f5d92858700 1 -- 172.21.15.88:0/1130538240 shutdown_connections 2023-07-14T17:44:25.007 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.023+0000 7f5d92858700 1 -- 172.21.15.88:0/1130538240 wait complete. 2023-07-14T17:44:25.017 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T17:44:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.159+0000 7f7f84b07700 1 -- 172.21.15.88:0/2851848959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f800ff710 msgr2=0x7f7f80106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.159+0000 7f7f84b07700 1 --2- 172.21.15.88:0/2851848959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f800ff710 0x7f7f80106fe0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7f7800b0a0 tx=0x7f7f7800b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.159+0000 7f7f84b07700 1 -- 172.21.15.88:0/2851848959 shutdown_connections 2023-07-14T17:44:25.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.159+0000 7f7f84b07700 1 --2- 172.21.15.88:0/2851848959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f800ff710 0x7f7f80106fe0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.159+0000 7f7f84b07700 1 --2- 172.21.15.88:0/2851848959 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f801000e0 0x7f7f801004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.159+0000 7f7f84b07700 1 --2- 172.21.15.88:0/2851848959 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f800fed80 0x7f7f800ff1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.159+0000 7f7f84b07700 1 -- 172.21.15.88:0/2851848959 >> 172.21.15.88:0/2851848959 conn(0x7f7f800f98a0 msgr2=0x7f7f800fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:25.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.159+0000 7f7f84b07700 1 -- 172.21.15.88:0/2851848959 shutdown_connections 2023-07-14T17:44:25.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.159+0000 7f7f84b07700 1 -- 172.21.15.88:0/2851848959 wait complete. 2023-07-14T17:44:25.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.160+0000 7f7f84b07700 1 Processor -- start 2023-07-14T17:44:25.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.160+0000 7f7f84b07700 1 -- start start 2023-07-14T17:44:25.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.160+0000 7f7f84b07700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f800fed80 0x7f7f80194900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.160+0000 7f7f84b07700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f800ff710 0x7f7f80194e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.160+0000 7f7f84b07700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f801000e0 0x7f7f8019beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.160+0000 7f7f84b07700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8019c480 con 0x7f7f800fed80 2023-07-14T17:44:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.160+0000 7f7f84b07700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8019c5c0 con 0x7f7f800ff710 2023-07-14T17:44:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.160+0000 7f7f84b07700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8019c750 con 0x7f7f801000e0 2023-07-14T17:44:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.160+0000 7f7f7e59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f800fed80 0x7f7f80194900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:25.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.160+0000 7f7f7e59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f800fed80 0x7f7f80194900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43030/0 (socket says 172.21.15.88:43030) 2023-07-14T17:44:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.160+0000 7f7f7e59c700 1 -- 172.21.15.88:0/2539797294 learned_addr learned my addr 172.21.15.88:0/2539797294 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.161+0000 7f7f7ed9d700 1 --2- 172.21.15.88:0/2539797294 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f801000e0 0x7f7f8019beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.161+0000 7f7f7dd9b700 1 --2- 172.21.15.88:0/2539797294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f800ff710 0x7f7f80194e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.161+0000 7f7f7e59c700 1 -- 172.21.15.88:0/2539797294 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f801000e0 msgr2=0x7f7f8019beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.161+0000 7f7f7e59c700 1 --2- 172.21.15.88:0/2539797294 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f801000e0 0x7f7f8019beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.161+0000 7f7f7e59c700 1 -- 172.21.15.88:0/2539797294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f800ff710 msgr2=0x7f7f80194e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.161+0000 7f7f7e59c700 1 --2- 172.21.15.88:0/2539797294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f800ff710 0x7f7f80194e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.161+0000 7f7f7e59c700 1 -- 172.21.15.88:0/2539797294 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f78009d00 con 0x7f7f800fed80 2023-07-14T17:44:25.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.161+0000 7f7f7ed9d700 1 --2- 172.21.15.88:0/2539797294 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f801000e0 0x7f7f8019beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:25.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.161+0000 7f7f7e59c700 1 --2- 172.21.15.88:0/2539797294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f800fed80 0x7f7f80194900 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7f7400d980 tx=0x7f7f7400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:25.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.161+0000 7f7f6f7fe700 1 -- 172.21.15.88:0/2539797294 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f7400de60 con 0x7f7f800fed80 2023-07-14T17:44:25.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.161+0000 7f7f6f7fe700 1 -- 172.21.15.88:0/2539797294 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7f740044b0 con 0x7f7f800fed80 2023-07-14T17:44:25.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.161+0000 7f7f84b07700 1 -- 172.21.15.88:0/2539797294 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f8019ca30 con 0x7f7f800fed80 2023-07-14T17:44:25.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.161+0000 7f7f6f7fe700 1 -- 172.21.15.88:0/2539797294 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f74003b40 con 0x7f7f800fed80 2023-07-14T17:44:25.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.161+0000 7f7f84b07700 1 -- 172.21.15.88:0/2539797294 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f8019cf40 con 0x7f7f800fed80 2023-07-14T17:44:25.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.162+0000 7f7f84b07700 1 -- 172.21.15.88:0/2539797294 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f80106180 con 0x7f7f800fed80 2023-07-14T17:44:25.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.163+0000 7f7f6f7fe700 1 -- 172.21.15.88:0/2539797294 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7f74004870 con 0x7f7f800fed80 2023-07-14T17:44:25.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.164+0000 7f7f6f7fe700 1 --2- 172.21.15.88:0/2539797294 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7f680a1150 0x7f7f680a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:25.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.164+0000 7f7f7dd9b700 1 --2- 172.21.15.88:0/2539797294 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7f680a1150 0x7f7f680a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:25.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.164+0000 7f7f6f7fe700 1 -- 172.21.15.88:0/2539797294 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7f740ccc70 con 0x7f7f800fed80 2023-07-14T17:44:25.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.164+0000 7f7f7dd9b700 1 --2- 172.21.15.88:0/2539797294 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7f680a1150 0x7f7f680a3600 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7f70005fd0 tx=0x7f7f70005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:25.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.167+0000 7f7f6f7fe700 1 -- 172.21.15.88:0/2539797294 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7f74096050 con 0x7f7f800fed80 2023-07-14T17:44:25.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.356+0000 7f7f84b07700 1 -- 172.21.15.88:0/2539797294 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7f8002cb50 con 0x7f7f800fed80 2023-07-14T17:44:25.339 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.357+0000 7f7f6f7fe700 1 -- 172.21.15.88:0/2539797294 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7f74009940 con 0x7f7f800fed80 2023-07-14T17:44:25.340 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-07-14T17:44:25.341 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.359+0000 7f7f84b07700 1 -- 172.21.15.88:0/2539797294 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7f680a1150 msgr2=0x7f7f680a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:25.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.359+0000 7f7f84b07700 1 --2- 172.21.15.88:0/2539797294 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7f680a1150 0x7f7f680a3600 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f7f70005fd0 tx=0x7f7f70005f60 comp rx=0 tx=0).stop 2023-07-14T17:44:25.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.359+0000 7f7f84b07700 1 -- 172.21.15.88:0/2539797294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f800fed80 msgr2=0x7f7f80194900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:25.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.359+0000 7f7f84b07700 1 --2- 172.21.15.88:0/2539797294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f800fed80 0x7f7f80194900 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7f7400d980 tx=0x7f7f7400dd40 comp rx=0 tx=0).stop 2023-07-14T17:44:25.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.359+0000 7f7f84b07700 1 -- 172.21.15.88:0/2539797294 shutdown_connections 2023-07-14T17:44:25.342 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.359+0000 7f7f84b07700 1 --2- 172.21.15.88:0/2539797294 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f7f801000e0 0x7f7f8019beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.359+0000 7f7f84b07700 1 --2- 172.21.15.88:0/2539797294 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f7f680a1150 0x7f7f680a3600 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.359+0000 7f7f84b07700 1 --2- 172.21.15.88:0/2539797294 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7f800fed80 0x7f7f80194900 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.359+0000 7f7f84b07700 1 --2- 172.21.15.88:0/2539797294 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f7f800ff710 0x7f7f80194e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.359+0000 7f7f84b07700 1 -- 172.21.15.88:0/2539797294 >> 172.21.15.88:0/2539797294 conn(0x7f7f800f98a0 msgr2=0x7f7f801032c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:25.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.359+0000 7f7f84b07700 1 -- 172.21.15.88:0/2539797294 shutdown_connections 2023-07-14T17:44:25.343 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.359+0000 7f7f84b07700 1 -- 172.21.15.88:0/2539797294 wait complete. 2023-07-14T17:44:25.352 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:25 smithi088 ceph-mon[127465]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 596 B/s wr, 10 op/s 2023-07-14T17:44:25.353 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:25 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1530593414' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:25.354 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T17:44:25.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:25 smithi178 ceph-mon[129368]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 596 B/s wr, 10 op/s 2023-07-14T17:44:25.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:25 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1530593414' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:25.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:25 smithi107 ceph-mon[128999]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 596 B/s wr, 10 op/s 2023-07-14T17:44:25.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:25 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1530593414' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:25.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.495+0000 7ff5252e1700 1 -- 172.21.15.88:0/4268500382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5200ffdb0 msgr2=0x7ff520100220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:25.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.495+0000 7ff5252e1700 1 --2- 172.21.15.88:0/4268500382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5200ffdb0 0x7ff520100220 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff510009990 tx=0x7ff510009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.496+0000 7ff5252e1700 1 -- 172.21.15.88:0/4268500382 shutdown_connections 2023-07-14T17:44:25.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.496+0000 7ff5252e1700 1 --2- 172.21.15.88:0/4268500382 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff520100760 0x7ff520107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.496+0000 7ff5252e1700 1 --2- 172.21.15.88:0/4268500382 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff5200feb20 0x7ff5200fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.496+0000 7ff5252e1700 1 --2- 172.21.15.88:0/4268500382 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5200ffdb0 0x7ff520100220 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.496+0000 7ff5252e1700 1 -- 172.21.15.88:0/4268500382 >> 172.21.15.88:0/4268500382 conn(0x7ff5200fa120 msgr2=0x7ff5200fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:25.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.496+0000 7ff5252e1700 1 -- 172.21.15.88:0/4268500382 shutdown_connections 2023-07-14T17:44:25.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.496+0000 7ff5252e1700 1 -- 172.21.15.88:0/4268500382 wait complete. 2023-07-14T17:44:25.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.496+0000 7ff5252e1700 1 Processor -- start 2023-07-14T17:44:25.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.496+0000 7ff5252e1700 1 -- start start 2023-07-14T17:44:25.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff5252e1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5200feb20 0x7ff5200694e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:25.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff5252e1700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5200ffdb0 0x7ff52019a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff5252e1700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff520100760 0x7ff52019cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff5252e1700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52019d3b0 con 0x7ff5200feb20 2023-07-14T17:44:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff5252e1700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52019d520 con 0x7ff5200ffdb0 2023-07-14T17:44:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff5252e1700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52019d6e0 con 0x7ff520100760 2023-07-14T17:44:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff51effd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5200feb20 0x7ff5200694e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff51effd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5200feb20 0x7ff5200694e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43044/0 (socket says 172.21.15.88:43044) 2023-07-14T17:44:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff51effd700 1 -- 172.21.15.88:0/3611119055 learned_addr learned my addr 172.21.15.88:0/3611119055 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:25.482 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff51f7fe700 1 --2- 172.21.15.88:0/3611119055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff520100760 0x7ff52019cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff51e7fc700 1 --2- 172.21.15.88:0/3611119055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5200ffdb0 0x7ff52019a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff51effd700 1 -- 172.21.15.88:0/3611119055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff520100760 msgr2=0x7ff52019cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff51effd700 1 --2- 172.21.15.88:0/3611119055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff520100760 0x7ff52019cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff51effd700 1 -- 172.21.15.88:0/3611119055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5200ffdb0 msgr2=0x7ff52019a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff51effd700 1 --2- 172.21.15.88:0/3611119055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5200ffdb0 0x7ff52019a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff51effd700 1 -- 172.21.15.88:0/3611119055 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5100095f0 con 0x7ff5200feb20 2023-07-14T17:44:25.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff51e7fc700 1 --2- 172.21.15.88:0/3611119055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5200ffdb0 0x7ff52019a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:25.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff51f7fe700 1 --2- 172.21.15.88:0/3611119055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff520100760 0x7ff52019cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:25.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff51effd700 1 --2- 172.21.15.88:0/3611119055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5200feb20 0x7ff5200694e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff51400d950 tx=0x7ff51400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:25.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff507fff700 1 -- 172.21.15.88:0/3611119055 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff514009b40 con 0x7ff5200feb20 2023-07-14T17:44:25.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff507fff700 1 -- 172.21.15.88:0/3611119055 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff514009ca0 con 0x7ff5200feb20 2023-07-14T17:44:25.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff5252e1700 1 -- 172.21.15.88:0/3611119055 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff52019d9c0 con 0x7ff5200feb20 2023-07-14T17:44:25.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff507fff700 1 -- 172.21.15.88:0/3611119055 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51400f620 con 0x7ff5200feb20 2023-07-14T17:44:25.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.497+0000 7ff5252e1700 1 -- 172.21.15.88:0/3611119055 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff52019ded0 con 0x7ff5200feb20 2023-07-14T17:44:25.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.498+0000 7ff5252e1700 1 -- 172.21.15.88:0/3611119055 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff520105b40 con 0x7ff5200feb20 2023-07-14T17:44:25.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.500+0000 7ff507fff700 1 -- 172.21.15.88:0/3611119055 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff514010460 con 0x7ff5200feb20 2023-07-14T17:44:25.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.500+0000 7ff507fff700 1 --2- 172.21.15.88:0/3611119055 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff5080a1150 0x7ff5080a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:25.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.501+0000 7ff51e7fc700 1 --2- 172.21.15.88:0/3611119055 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff5080a1150 0x7ff5080a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:25.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.501+0000 7ff507fff700 1 -- 172.21.15.88:0/3611119055 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff5140ccf90 con 0x7ff5200feb20 2023-07-14T17:44:25.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.501+0000 7ff51e7fc700 1 --2- 172.21.15.88:0/3611119055 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff5080a1150 0x7ff5080a3600 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff51000b2e0 tx=0x7ff510005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:25.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.504+0000 7ff507fff700 1 -- 172.21.15.88:0/3611119055 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff5140963d0 con 0x7ff5200feb20 2023-07-14T17:44:25.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.694+0000 7ff5252e1700 1 -- 172.21.15.88:0/3611119055 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff52002cb50 con 0x7ff5200feb20 2023-07-14T17:44:25.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.695+0000 7ff507fff700 1 -- 172.21.15.88:0/3611119055 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff514016020 con 0x7ff5200feb20 2023-07-14T17:44:25.677 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 64 2023-07-14T17:44:25.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.697+0000 7ff5252e1700 1 -- 172.21.15.88:0/3611119055 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff5080a1150 msgr2=0x7ff5080a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:25.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.697+0000 7ff5252e1700 1 --2- 172.21.15.88:0/3611119055 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff5080a1150 0x7ff5080a3600 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff51000b2e0 tx=0x7ff510005fd0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.697+0000 7ff5252e1700 1 -- 172.21.15.88:0/3611119055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5200feb20 msgr2=0x7ff5200694e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:25.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.697+0000 7ff5252e1700 1 --2- 172.21.15.88:0/3611119055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5200feb20 0x7ff5200694e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ff51400d950 tx=0x7ff51400dd10 comp rx=0 tx=0).stop 2023-07-14T17:44:25.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.697+0000 7ff5252e1700 1 -- 172.21.15.88:0/3611119055 shutdown_connections 2023-07-14T17:44:25.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.697+0000 7ff5252e1700 1 --2- 172.21.15.88:0/3611119055 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff520100760 0x7ff52019cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.697+0000 7ff5252e1700 1 --2- 172.21.15.88:0/3611119055 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff5080a1150 0x7ff5080a3600 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.697+0000 7ff5252e1700 1 --2- 172.21.15.88:0/3611119055 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff5200feb20 0x7ff5200694e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.697+0000 7ff5252e1700 1 --2- 172.21.15.88:0/3611119055 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff5200ffdb0 0x7ff52019a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.697+0000 7ff5252e1700 1 -- 172.21.15.88:0/3611119055 >> 172.21.15.88:0/3611119055 conn(0x7ff5200fa120 msgr2=0x7ff5200fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:25.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.697+0000 7ff5252e1700 1 -- 172.21.15.88:0/3611119055 shutdown_connections 2023-07-14T17:44:25.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.697+0000 7ff5252e1700 1 -- 172.21.15.88:0/3611119055 wait complete. 2023-07-14T17:44:25.691 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T17:44:25.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.835+0000 7f20da8ee700 1 -- 172.21.15.88:0/1079824868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20d40ffd00 msgr2=0x7f20d4100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:25.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.835+0000 7f20da8ee700 1 --2- 172.21.15.88:0/1079824868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20d40ffd00 0x7f20d4100150 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f20c8009990 tx=0x7f20c8009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.835+0000 7f20da8ee700 1 -- 172.21.15.88:0/1079824868 shutdown_connections 2023-07-14T17:44:25.818 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.835+0000 7f20da8ee700 1 --2- 172.21.15.88:0/1079824868 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20d4100690 0x7f20d4106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.835+0000 7f20da8ee700 1 --2- 172.21.15.88:0/1079824868 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d40feb10 0x7f20d40fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.835+0000 7f20da8ee700 1 --2- 172.21.15.88:0/1079824868 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20d40ffd00 0x7f20d4100150 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.835+0000 7f20da8ee700 1 -- 172.21.15.88:0/1079824868 >> 172.21.15.88:0/1079824868 conn(0x7f20d40fa130 msgr2=0x7f20d40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:25.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.835+0000 7f20da8ee700 1 -- 172.21.15.88:0/1079824868 shutdown_connections 2023-07-14T17:44:25.819 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.835+0000 7f20da8ee700 1 -- 172.21.15.88:0/1079824868 wait complete. 2023-07-14T17:44:25.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.836+0000 7f20da8ee700 1 Processor -- start 2023-07-14T17:44:25.821 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.836+0000 7f20da8ee700 1 -- start start 2023-07-14T17:44:25.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.836+0000 7f20da8ee700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d40feb10 0x7f20d4194780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:25.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.836+0000 7f20da8ee700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20d40ffd00 0x7f20d4194cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:25.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.836+0000 7f20da8ee700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20d4100690 0x7f20d419bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:25.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.836+0000 7f20da8ee700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d419c300 con 0x7f20d40ffd00 2023-07-14T17:44:25.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.836+0000 7f20da8ee700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d419c440 con 0x7f20d40feb10 2023-07-14T17:44:25.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.836+0000 7f20da8ee700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d419c5d0 con 0x7f20d4100690 2023-07-14T17:44:25.822 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.836+0000 7f20d37fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20d40ffd00 0x7f20d4194cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:25.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.836+0000 7f20d37fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20d40ffd00 0x7f20d4194cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43060/0 (socket says 172.21.15.88:43060) 2023-07-14T17:44:25.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.836+0000 7f20d37fe700 1 -- 172.21.15.88:0/3791849021 learned_addr learned my addr 172.21.15.88:0/3791849021 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:25.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.837+0000 7f20d3fff700 1 --2- 172.21.15.88:0/3791849021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d40feb10 0x7f20d4194780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:25.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.837+0000 7f20d8e8b700 1 --2- 172.21.15.88:0/3791849021 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20d4100690 0x7f20d419bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:25.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.837+0000 7f20d37fe700 1 -- 172.21.15.88:0/3791849021 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20d4100690 msgr2=0x7f20d419bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:25.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.837+0000 7f20d37fe700 1 --2- 172.21.15.88:0/3791849021 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20d4100690 0x7f20d419bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.837+0000 7f20d37fe700 1 -- 172.21.15.88:0/3791849021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d40feb10 msgr2=0x7f20d4194780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:25.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.837+0000 7f20d37fe700 1 --2- 172.21.15.88:0/3791849021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d40feb10 0x7f20d4194780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:25.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.837+0000 7f20d37fe700 1 -- 172.21.15.88:0/3791849021 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c80095f0 con 0x7f20d40ffd00 2023-07-14T17:44:25.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.837+0000 7f20d37fe700 1 --2- 172.21.15.88:0/3791849021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20d40ffd00 0x7f20d4194cc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f20c80095c0 tx=0x7f20c8005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:25.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.837+0000 7f20d17fa700 1 -- 172.21.15.88:0/3791849021 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c801d070 con 0x7f20d40ffd00 2023-07-14T17:44:25.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.837+0000 7f20da8ee700 1 -- 172.21.15.88:0/3791849021 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d419c850 con 0x7f20d40ffd00 2023-07-14T17:44:25.824 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.837+0000 7f20d17fa700 1 -- 172.21.15.88:0/3791849021 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f20c8004000 con 0x7f20d40ffd00 2023-07-14T17:44:25.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.837+0000 7f20d17fa700 1 -- 172.21.15.88:0/3791849021 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c8022cd0 con 0x7f20d40ffd00 2023-07-14T17:44:25.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.837+0000 7f20da8ee700 1 -- 172.21.15.88:0/3791849021 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d419cc80 con 0x7f20d40ffd00 2023-07-14T17:44:25.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.838+0000 7f20da8ee700 1 -- 172.21.15.88:0/3791849021 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d4105af0 con 0x7f20d40ffd00 2023-07-14T17:44:25.825 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.839+0000 7f20d17fa700 1 -- 172.21.15.88:0/3791849021 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f20c8022470 con 0x7f20d40ffd00 2023-07-14T17:44:25.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.839+0000 7f20d17fa700 1 --2- 172.21.15.88:0/3791849021 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f20bc0a11d0 0x7f20bc0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:25.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.840+0000 7f20d3fff700 1 --2- 172.21.15.88:0/3791849021 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f20bc0a11d0 0x7f20bc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:25.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.840+0000 7f20d17fa700 1 -- 172.21.15.88:0/3791849021 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f20c80ce200 con 0x7f20d40ffd00 2023-07-14T17:44:25.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.840+0000 7f20d3fff700 1 --2- 172.21.15.88:0/3791849021 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f20bc0a11d0 0x7f20bc0a3680 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f20d40ffb60 tx=0x7f20c400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:25.826 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:25.844+0000 7f20d17fa700 1 -- 172.21.15.88:0/3791849021 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f20c8097670 con 0x7f20d40ffd00 2023-07-14T17:44:26.021 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.039+0000 7f20da8ee700 1 -- 172.21.15.88:0/3791849021 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f20d4061250 con 0x7f20d40ffd00 2023-07-14T17:44:26.022 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.039+0000 7f20d17fa700 1 -- 172.21.15.88:0/3791849021 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7f20c809ac90 con 0x7f20d40ffd00 2023-07-14T17:44:26.022 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-07-14T17:44:26.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.041+0000 7f20da8ee700 1 -- 172.21.15.88:0/3791849021 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f20bc0a11d0 msgr2=0x7f20bc0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:26.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.041+0000 7f20da8ee700 1 --2- 172.21.15.88:0/3791849021 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f20bc0a11d0 0x7f20bc0a3680 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f20d40ffb60 tx=0x7f20c400b410 comp rx=0 tx=0).stop 2023-07-14T17:44:26.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.041+0000 7f20da8ee700 1 -- 172.21.15.88:0/3791849021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20d40ffd00 msgr2=0x7f20d4194cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:26.024 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.041+0000 7f20da8ee700 1 --2- 172.21.15.88:0/3791849021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20d40ffd00 0x7f20d4194cc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f20c80095c0 tx=0x7f20c8005740 comp rx=0 tx=0).stop 2023-07-14T17:44:26.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.041+0000 7f20da8ee700 1 -- 172.21.15.88:0/3791849021 shutdown_connections 2023-07-14T17:44:26.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.041+0000 7f20da8ee700 1 --2- 172.21.15.88:0/3791849021 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f20d4100690 0x7f20d419bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.041+0000 7f20da8ee700 1 --2- 172.21.15.88:0/3791849021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f20d40feb10 0x7f20d4194780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.041+0000 7f20da8ee700 1 --2- 172.21.15.88:0/3791849021 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f20bc0a11d0 0x7f20bc0a3680 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.041+0000 7f20da8ee700 1 --2- 172.21.15.88:0/3791849021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20d40ffd00 0x7f20d4194cc0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.041+0000 7f20da8ee700 1 -- 172.21.15.88:0/3791849021 >> 172.21.15.88:0/3791849021 conn(0x7f20d40fa130 msgr2=0x7f20d40fc470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:26.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.041+0000 7f20da8ee700 1 -- 172.21.15.88:0/3791849021 shutdown_connections 2023-07-14T17:44:26.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.042+0000 7f20da8ee700 1 -- 172.21.15.88:0/3791849021 wait complete. 2023-07-14T17:44:26.036 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:44:26.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.175+0000 7f79455a3700 1 -- 172.21.15.88:0/1481547649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79401006d0 msgr2=0x7f7940106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:26.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.175+0000 7f79455a3700 1 --2- 172.21.15.88:0/1481547649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79401006d0 0x7f7940106f90 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f793800b0a0 tx=0x7f793800b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.175+0000 7f79455a3700 1 -- 172.21.15.88:0/1481547649 shutdown_connections 2023-07-14T17:44:26.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.175+0000 7f79455a3700 1 --2- 172.21.15.88:0/1481547649 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79401006d0 0x7f7940106f90 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.175+0000 7f79455a3700 1 --2- 172.21.15.88:0/1481547649 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f79400feb50 0x7f79400fef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.175+0000 7f79455a3700 1 --2- 172.21.15.88:0/1481547649 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79400ffd40 0x7f7940100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.175+0000 7f79455a3700 1 -- 172.21.15.88:0/1481547649 >> 172.21.15.88:0/1481547649 conn(0x7f79400fa150 msgr2=0x7f79400fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:26.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.175+0000 7f79455a3700 1 -- 172.21.15.88:0/1481547649 shutdown_connections 2023-07-14T17:44:26.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.175+0000 7f79455a3700 1 -- 172.21.15.88:0/1481547649 wait complete. 2023-07-14T17:44:26.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.176+0000 7f79455a3700 1 Processor -- start 2023-07-14T17:44:26.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.176+0000 7f79455a3700 1 -- start start 2023-07-14T17:44:26.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f79455a3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79400feb50 0x7f7940194800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:26.161 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f79455a3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79400ffd40 0x7f7940194d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:26.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f79455a3700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f79401006d0 0x7f794019bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:26.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f79455a3700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794019c380 con 0x7f79400ffd40 2023-07-14T17:44:26.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f79455a3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794019c4c0 con 0x7f79400feb50 2023-07-14T17:44:26.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f79455a3700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794019c650 con 0x7f79401006d0 2023-07-14T17:44:26.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f793e7fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79400ffd40 0x7f7940194d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:26.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f793e7fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79400ffd40 0x7f7940194d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43072/0 (socket says 172.21.15.88:43072) 2023-07-14T17:44:26.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f793e7fc700 1 -- 172.21.15.88:0/340866552 learned_addr learned my addr 172.21.15.88:0/340866552 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:26.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f793f7fe700 1 --2- 172.21.15.88:0/340866552 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f79401006d0 0x7f794019bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:26.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f793effd700 1 --2- 172.21.15.88:0/340866552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79400feb50 0x7f7940194800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:26.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f793e7fc700 1 -- 172.21.15.88:0/340866552 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f79401006d0 msgr2=0x7f794019bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:26.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f793e7fc700 1 --2- 172.21.15.88:0/340866552 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f79401006d0 0x7f794019bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f793e7fc700 1 -- 172.21.15.88:0/340866552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79400feb50 msgr2=0x7f7940194800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:26.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f793e7fc700 1 --2- 172.21.15.88:0/340866552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79400feb50 0x7f7940194800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f793e7fc700 1 -- 172.21.15.88:0/340866552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7938009d00 con 0x7f79400ffd40 2023-07-14T17:44:26.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f793e7fc700 1 --2- 172.21.15.88:0/340866552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79400ffd40 0x7f7940194d40 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f793400d980 tx=0x7f793400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:26.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f7927fff700 1 -- 172.21.15.88:0/340866552 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7934004220 con 0x7f79400ffd40 2023-07-14T17:44:26.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f7927fff700 1 -- 172.21.15.88:0/340866552 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7934007640 con 0x7f79400ffd40 2023-07-14T17:44:26.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f79455a3700 1 -- 172.21.15.88:0/340866552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794019c930 con 0x7f79400ffd40 2023-07-14T17:44:26.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f7927fff700 1 -- 172.21.15.88:0/340866552 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79340055e0 con 0x7f79400ffd40 2023-07-14T17:44:26.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.177+0000 7f79455a3700 1 -- 172.21.15.88:0/340866552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794019ce40 con 0x7f79400ffd40 2023-07-14T17:44:26.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.178+0000 7f79455a3700 1 -- 172.21.15.88:0/340866552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794010a1e0 con 0x7f79400ffd40 2023-07-14T17:44:26.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.179+0000 7f7927fff700 1 -- 172.21.15.88:0/340866552 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7934004750 con 0x7f79400ffd40 2023-07-14T17:44:26.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.180+0000 7f7927fff700 1 --2- 172.21.15.88:0/340866552 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f79280a11a0 0x7f79280a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:26.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.180+0000 7f793effd700 1 --2- 172.21.15.88:0/340866552 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f79280a11a0 0x7f79280a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:26.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.180+0000 7f7927fff700 1 -- 172.21.15.88:0/340866552 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f79340ccf80 con 0x7f79400ffd40 2023-07-14T17:44:26.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.180+0000 7f793effd700 1 --2- 172.21.15.88:0/340866552 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f79280a11a0 0x7f79280a3650 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7930005950 tx=0x7f79300058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:26.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.183+0000 7f7927fff700 1 -- 172.21.15.88:0/340866552 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7934096360 con 0x7f79400ffd40 2023-07-14T17:44:26.397 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.415+0000 7f79455a3700 1 -- 172.21.15.88:0/340866552 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f794010a3d0 con 0x7f79400ffd40 2023-07-14T17:44:26.398 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.416+0000 7f7927fff700 1 -- 172.21.15.88:0/340866552 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10885 (secure 0 0 0) 0x7f7934016020 con 0x7f79400ffd40 2023-07-14T17:44:26.398 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:26.398 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T17:42:42.314442+0000","modified":"2023-07-14T17:43:04.774381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24473,"mds_2":24469,"mds_3":24454,"mds_4":24449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6839/3046592155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3046592155},{"type":"v1","addr":"172.21.15.88:6839","nonce":3046592155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6841/2509502918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6840","nonce":2509502918},{"type":"v1","addr":"172.21.15.88:6841","nonce":2509502918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6835/2332025068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":2332025068},{"type":"v1","addr":"172.21.15.178:6835","nonce":2332025068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24449":{"gid":24449,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.107:6835/3062637672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3062637672},{"type":"v1","addr":"172.21.15.107:6835","nonce":3062637672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.178:6837/3950404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3950404084},{"type":"v1","addr":"172.21.15.178:6837","nonce":3950404084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.178:6839/2953415077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2953415077},{"type":"v1","addr":"172.21.15.178:6839","nonce":2953415077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/1152608615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1152608615},{"type":"v1","addr":"172.21.15.107:6837","nonce":1152608615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6839/31646548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":31646548},{"type":"v1","addr":"172.21.15.107:6839","nonce":31646548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T17:44:26.400 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.418+0000 7f79455a3700 1 -- 172.21.15.88:0/340866552 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f79280a11a0 msgr2=0x7f79280a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:26.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.418+0000 7f79455a3700 1 --2- 172.21.15.88:0/340866552 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f79280a11a0 0x7f79280a3650 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7930005950 tx=0x7f79300058e0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.418+0000 7f79455a3700 1 -- 172.21.15.88:0/340866552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79400ffd40 msgr2=0x7f7940194d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:26.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.418+0000 7f79455a3700 1 --2- 172.21.15.88:0/340866552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79400ffd40 0x7f7940194d40 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f793400d980 tx=0x7f793400dc90 comp rx=0 tx=0).stop 2023-07-14T17:44:26.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.418+0000 7f79455a3700 1 -- 172.21.15.88:0/340866552 shutdown_connections 2023-07-14T17:44:26.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.418+0000 7f79455a3700 1 --2- 172.21.15.88:0/340866552 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f79401006d0 0x7f794019bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.401 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.418+0000 7f79455a3700 1 --2- 172.21.15.88:0/340866552 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f79400feb50 0x7f7940194800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.418+0000 7f79455a3700 1 --2- 172.21.15.88:0/340866552 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f79280a11a0 0x7f79280a3650 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.418+0000 7f79455a3700 1 --2- 172.21.15.88:0/340866552 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f79400ffd40 0x7f7940194d40 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.418+0000 7f79455a3700 1 -- 172.21.15.88:0/340866552 >> 172.21.15.88:0/340866552 conn(0x7f79400fa150 msgr2=0x7f79400fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:26.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.418+0000 7f79455a3700 1 -- 172.21.15.88:0/340866552 shutdown_connections 2023-07-14T17:44:26.402 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.418+0000 7f79455a3700 1 -- 172.21.15.88:0/340866552 wait complete. 2023-07-14T17:44:26.403 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 27 2023-07-14T17:44:26.411 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:26 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1130538240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:44:26.412 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:26 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2539797294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:44:26.412 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:26 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3611119055' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:44:26.413 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:44:26.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:26 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1130538240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:44:26.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:26 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2539797294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:44:26.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:26 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3611119055' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:44:26.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:26 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1130538240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:44:26.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:26 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2539797294' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:44:26.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:26 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3611119055' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:44:26.538 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.555+0000 7fd40b6e9700 1 -- 172.21.15.88:0/3021800021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4040ffcf0 msgr2=0x7fd404100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:26.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.555+0000 7fd40b6e9700 1 --2- 172.21.15.88:0/3021800021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4040ffcf0 0x7fd404100140 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd3f4009990 tx=0x7fd3f4009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.555+0000 7fd40b6e9700 1 -- 172.21.15.88:0/3021800021 shutdown_connections 2023-07-14T17:44:26.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.555+0000 7fd40b6e9700 1 --2- 172.21.15.88:0/3021800021 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd404100680 0x7fd404106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.555+0000 7fd40b6e9700 1 --2- 172.21.15.88:0/3021800021 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4040feb00 0x7fd4040fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.555+0000 7fd40b6e9700 1 --2- 172.21.15.88:0/3021800021 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4040ffcf0 0x7fd404100140 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.539 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.555+0000 7fd40b6e9700 1 -- 172.21.15.88:0/3021800021 >> 172.21.15.88:0/3021800021 conn(0x7fd4040fa100 msgr2=0x7fd4040fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:26.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.555+0000 7fd40b6e9700 1 -- 172.21.15.88:0/3021800021 shutdown_connections 2023-07-14T17:44:26.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.555+0000 7fd40b6e9700 1 -- 172.21.15.88:0/3021800021 wait complete. 2023-07-14T17:44:26.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.555+0000 7fd40b6e9700 1 Processor -- start 2023-07-14T17:44:26.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.555+0000 7fd40b6e9700 1 -- start start 2023-07-14T17:44:26.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.555+0000 7fd40b6e9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4040feb00 0x7fd404195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:26.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.555+0000 7fd40b6e9700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4040ffcf0 0x7fd404195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:26.540 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.555+0000 7fd40b6e9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd404100680 0x7fd40419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.555+0000 7fd40b6e9700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40419d380 con 0x7fd404100680 2023-07-14T17:44:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.555+0000 7fd40b6e9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40419d4c0 con 0x7fd4040feb00 2023-07-14T17:44:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.556+0000 7fd40b6e9700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40419d650 con 0x7fd4040ffcf0 2023-07-14T17:44:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.556+0000 7fd409c86700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd404100680 0x7fd40419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.556+0000 7fd409c86700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd404100680 0x7fd40419cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43088/0 (socket says 172.21.15.88:43088) 2023-07-14T17:44:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.556+0000 7fd409485700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4040feb00 0x7fd404195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.556+0000 7fd409c86700 1 -- 172.21.15.88:0/4220777174 learned_addr learned my addr 172.21.15.88:0/4220777174 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:26.541 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.556+0000 7fd408c84700 1 --2- 172.21.15.88:0/4220777174 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4040ffcf0 0x7fd404195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:26.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.556+0000 7fd409c86700 1 -- 172.21.15.88:0/4220777174 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4040ffcf0 msgr2=0x7fd404195d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:26.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.556+0000 7fd409c86700 1 --2- 172.21.15.88:0/4220777174 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4040ffcf0 0x7fd404195d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.556+0000 7fd409c86700 1 -- 172.21.15.88:0/4220777174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4040feb00 msgr2=0x7fd404195800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:26.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.556+0000 7fd409c86700 1 --2- 172.21.15.88:0/4220777174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4040feb00 0x7fd404195800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.556+0000 7fd409c86700 1 -- 172.21.15.88:0/4220777174 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f40095f0 con 0x7fd404100680 2023-07-14T17:44:26.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.556+0000 7fd409c86700 1 --2- 172.21.15.88:0/4220777174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd404100680 0x7fd40419cdb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00ef30 tx=0x7fd3fc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:26.542 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.556+0000 7fd3fa7fc700 1 -- 172.21.15.88:0/4220777174 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc018070 con 0x7fd404100680 2023-07-14T17:44:26.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.556+0000 7fd40b6e9700 1 -- 172.21.15.88:0/4220777174 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd40419d8d0 con 0x7fd404100680 2023-07-14T17:44:26.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.556+0000 7fd3fa7fc700 1 -- 172.21.15.88:0/4220777174 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd3fc00f040 con 0x7fd404100680 2023-07-14T17:44:26.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.556+0000 7fd3fa7fc700 1 -- 172.21.15.88:0/4220777174 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc00cbc0 con 0x7fd404100680 2023-07-14T17:44:26.543 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.556+0000 7fd40b6e9700 1 -- 172.21.15.88:0/4220777174 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40419dd90 con 0x7fd404100680 2023-07-14T17:44:26.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.557+0000 7fd40b6e9700 1 -- 172.21.15.88:0/4220777174 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40410a180 con 0x7fd404100680 2023-07-14T17:44:26.544 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.558+0000 7fd3fa7fc700 1 -- 172.21.15.88:0/4220777174 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd3fc00cd80 con 0x7fd404100680 2023-07-14T17:44:26.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.559+0000 7fd3fa7fc700 1 --2- 172.21.15.88:0/4220777174 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd3f00a1220 0x7fd3f00a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:26.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.559+0000 7fd409485700 1 --2- 172.21.15.88:0/4220777174 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd3f00a1220 0x7fd3f00a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:26.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.559+0000 7fd3fa7fc700 1 -- 172.21.15.88:0/4220777174 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd3fc0ccd70 con 0x7fd404100680 2023-07-14T17:44:26.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.559+0000 7fd409485700 1 --2- 172.21.15.88:0/4220777174 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd3f00a1220 0x7fd3f00a36d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd4040ffb50 tx=0x7fd40000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:26.545 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.562+0000 7fd3fa7fc700 1 -- 172.21.15.88:0/4220777174 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd3fc0961e0 con 0x7fd404100680 2023-07-14T17:44:26.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.752+0000 7fd40b6e9700 1 -- 172.21.15.88:0/4220777174 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd40419e3d0 con 0x7fd404100680 2023-07-14T17:44:26.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.754+0000 7fd3fa7fc700 1 -- 172.21.15.88:0/4220777174 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24512 (secure 0 0 0) 0x7fd3fc099800 con 0x7fd404100680 2023-07-14T17:44:26.736 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:26.737 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":88,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:43:44.887959+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:44:26.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.756+0000 7fd40b6e9700 1 -- 172.21.15.88:0/4220777174 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd3f00a1220 msgr2=0x7fd3f00a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:26.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.756+0000 7fd40b6e9700 1 --2- 172.21.15.88:0/4220777174 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd3f00a1220 0x7fd3f00a36d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd4040ffb50 tx=0x7fd40000a380 comp rx=0 tx=0).stop 2023-07-14T17:44:26.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.756+0000 7fd40b6e9700 1 -- 172.21.15.88:0/4220777174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd404100680 msgr2=0x7fd40419cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.756+0000 7fd40b6e9700 1 --2- 172.21.15.88:0/4220777174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd404100680 0x7fd40419cdb0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00ef30 tx=0x7fd3fc00c3a0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.756+0000 7fd40b6e9700 1 -- 172.21.15.88:0/4220777174 shutdown_connections 2023-07-14T17:44:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.756+0000 7fd40b6e9700 1 --2- 172.21.15.88:0/4220777174 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd3f00a1220 0x7fd3f00a36d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.756+0000 7fd40b6e9700 1 --2- 172.21.15.88:0/4220777174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd404100680 0x7fd40419cdb0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.756+0000 7fd40b6e9700 1 --2- 172.21.15.88:0/4220777174 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4040feb00 0x7fd404195800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.756+0000 7fd40b6e9700 1 --2- 172.21.15.88:0/4220777174 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4040ffcf0 0x7fd404195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.756+0000 7fd40b6e9700 1 -- 172.21.15.88:0/4220777174 >> 172.21.15.88:0/4220777174 conn(0x7fd4040fa100 msgr2=0x7fd4040677a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:26.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.756+0000 7fd40b6e9700 1 -- 172.21.15.88:0/4220777174 shutdown_connections 2023-07-14T17:44:26.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.756+0000 7fd40b6e9700 1 -- 172.21.15.88:0/4220777174 wait complete. 2023-07-14T17:44:26.751 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T17:38:50.126535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T17:42:40.725870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T17:42:41.512749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T17:42:44.764573+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T17:44:26.752 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T17:44:26.876 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.894+0000 7f25e5f28700 1 -- 172.21.15.88:0/117113919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e0100ea0 msgr2=0x7f25e0101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:26.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.894+0000 7f25e5f28700 1 --2- 172.21.15.88:0/117113919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e0100ea0 0x7f25e0101310 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f25d0009990 tx=0x7f25d0009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.894+0000 7f25e5f28700 1 -- 172.21.15.88:0/117113919 shutdown_connections 2023-07-14T17:44:26.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.894+0000 7f25e5f28700 1 --2- 172.21.15.88:0/117113919 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25e0101850 0x7f25e01080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.894+0000 7f25e5f28700 1 --2- 172.21.15.88:0/117113919 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f25e00ffc10 0x7f25e0100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.894+0000 7f25e5f28700 1 --2- 172.21.15.88:0/117113919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e0100ea0 0x7f25e0101310 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.877 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.894+0000 7f25e5f28700 1 -- 172.21.15.88:0/117113919 >> 172.21.15.88:0/117113919 conn(0x7f25e00fb210 msgr2=0x7f25e00fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:26.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.894+0000 7f25e5f28700 1 -- 172.21.15.88:0/117113919 shutdown_connections 2023-07-14T17:44:26.878 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.894+0000 7f25e5f28700 1 -- 172.21.15.88:0/117113919 wait complete. 2023-07-14T17:44:26.879 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25e5f28700 1 Processor -- start 2023-07-14T17:44:26.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25e5f28700 1 -- start start 2023-07-14T17:44:26.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25e5f28700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e00ffc10 0x7f25e00743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:26.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25e5f28700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f25e0100ea0 0x7f25e0072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:26.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25e5f28700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25e0101850 0x7f25e0072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:26.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25e5f28700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e0073710 con 0x7f25e00ffc10 2023-07-14T17:44:26.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25e5f28700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e0073880 con 0x7f25e0101850 2023-07-14T17:44:26.880 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25e5f28700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e019f910 con 0x7f25e0100ea0 2023-07-14T17:44:26.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25df7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e00ffc10 0x7f25e00743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:26.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25df7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e00ffc10 0x7f25e00743f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43098/0 (socket says 172.21.15.88:43098) 2023-07-14T17:44:26.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25df7fe700 1 -- 172.21.15.88:0/1501241673 learned_addr learned my addr 172.21.15.88:0/1501241673 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:26.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25dffff700 1 --2- 172.21.15.88:0/1501241673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25e0101850 0x7f25e0072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:26.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25df7fe700 1 -- 172.21.15.88:0/1501241673 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f25e0100ea0 msgr2=0x7f25e0072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:26.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25df7fe700 1 --2- 172.21.15.88:0/1501241673 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f25e0100ea0 0x7f25e0072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.881 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25df7fe700 1 -- 172.21.15.88:0/1501241673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25e0101850 msgr2=0x7f25e0072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:26.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25df7fe700 1 --2- 172.21.15.88:0/1501241673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25e0101850 0x7f25e0072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:26.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25df7fe700 1 -- 172.21.15.88:0/1501241673 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d00095f0 con 0x7f25e00ffc10 2023-07-14T17:44:26.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25dffff700 1 --2- 172.21.15.88:0/1501241673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25e0101850 0x7f25e0072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:26.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.895+0000 7f25df7fe700 1 --2- 172.21.15.88:0/1501241673 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e00ffc10 0x7f25e00743f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f25c800d950 tx=0x7f25c800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:26.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.896+0000 7f25dcff9700 1 -- 172.21.15.88:0/1501241673 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c8009b40 con 0x7f25e00ffc10 2023-07-14T17:44:26.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.896+0000 7f25dcff9700 1 -- 172.21.15.88:0/1501241673 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f25c8009ca0 con 0x7f25e00ffc10 2023-07-14T17:44:26.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.896+0000 7f25e5f28700 1 -- 172.21.15.88:0/1501241673 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e019fbf0 con 0x7f25e00ffc10 2023-07-14T17:44:26.882 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.896+0000 7f25dcff9700 1 -- 172.21.15.88:0/1501241673 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c800f620 con 0x7f25e00ffc10 2023-07-14T17:44:26.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.896+0000 7f25e5f28700 1 -- 172.21.15.88:0/1501241673 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e01a0100 con 0x7f25e00ffc10 2023-07-14T17:44:26.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.897+0000 7f25e5f28700 1 -- 172.21.15.88:0/1501241673 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e01072f0 con 0x7f25e00ffc10 2023-07-14T17:44:26.883 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.898+0000 7f25dcff9700 1 -- 172.21.15.88:0/1501241673 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f25c8010460 con 0x7f25e00ffc10 2023-07-14T17:44:26.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.899+0000 7f25dcff9700 1 --2- 172.21.15.88:0/1501241673 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f25cc0a1150 0x7f25cc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:26.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.899+0000 7f25deffd700 1 --2- 172.21.15.88:0/1501241673 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f25cc0a1150 0x7f25cc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:26.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.899+0000 7f25dcff9700 1 -- 172.21.15.88:0/1501241673 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f25c80cd000 con 0x7f25e00ffc10 2023-07-14T17:44:26.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.899+0000 7f25deffd700 1 --2- 172.21.15.88:0/1501241673 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f25cc0a1150 0x7f25cc0a3600 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f25d0003f60 tx=0x7f25d0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:26.884 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:26.902+0000 7f25dcff9700 1 -- 172.21.15.88:0/1501241673 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f25c8096440 con 0x7f25e00ffc10 2023-07-14T17:44:27.073 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.091+0000 7f25e5f28700 1 -- 172.21.15.88:0/1501241673 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f25e002ca70 con 0x7f25e00ffc10 2023-07-14T17:44:27.074 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.091+0000 7f25dcff9700 1 -- 172.21.15.88:0/1501241673 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f25c80160a0 con 0x7f25e00ffc10 2023-07-14T17:44:27.074 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-07-14T17:44:27.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.093+0000 7f25e5f28700 1 -- 172.21.15.88:0/1501241673 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f25cc0a1150 msgr2=0x7f25cc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:27.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.093+0000 7f25e5f28700 1 --2- 172.21.15.88:0/1501241673 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f25cc0a1150 0x7f25cc0a3600 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f25d0003f60 tx=0x7f25d0005fb0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.093+0000 7f25e5f28700 1 -- 172.21.15.88:0/1501241673 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e00ffc10 msgr2=0x7f25e00743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:27.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.093+0000 7f25e5f28700 1 --2- 172.21.15.88:0/1501241673 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e00ffc10 0x7f25e00743f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f25c800d950 tx=0x7f25c800dd10 comp rx=0 tx=0).stop 2023-07-14T17:44:27.076 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.093+0000 7f25e5f28700 1 -- 172.21.15.88:0/1501241673 shutdown_connections 2023-07-14T17:44:27.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.093+0000 7f25e5f28700 1 --2- 172.21.15.88:0/1501241673 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f25e0101850 0x7f25e0072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.093+0000 7f25e5f28700 1 --2- 172.21.15.88:0/1501241673 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f25cc0a1150 0x7f25cc0a3600 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.093+0000 7f25e5f28700 1 --2- 172.21.15.88:0/1501241673 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f25e00ffc10 0x7f25e00743f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.093+0000 7f25e5f28700 1 --2- 172.21.15.88:0/1501241673 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f25e0100ea0 0x7f25e0072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.093+0000 7f25e5f28700 1 -- 172.21.15.88:0/1501241673 >> 172.21.15.88:0/1501241673 conn(0x7f25e00fb210 msgr2=0x7f25e0104430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:27.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.093+0000 7f25e5f28700 1 -- 172.21.15.88:0/1501241673 shutdown_connections 2023-07-14T17:44:27.077 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.093+0000 7f25e5f28700 1 -- 172.21.15.88:0/1501241673 wait complete. 2023-07-14T17:44:27.088 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T17:44:27.214 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.230+0000 7f9240c5f700 1 -- 172.21.15.88:0/260363032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f923c0ff710 msgr2=0x7f923c106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:27.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.230+0000 7f9240c5f700 1 --2- 172.21.15.88:0/260363032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f923c0ff710 0x7f923c106fc0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f923400b0a0 tx=0x7f923400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.230+0000 7f9240c5f700 1 -- 172.21.15.88:0/260363032 shutdown_connections 2023-07-14T17:44:27.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.230+0000 7f9240c5f700 1 --2- 172.21.15.88:0/260363032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f923c0ff710 0x7f923c106fc0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.230+0000 7f9240c5f700 1 --2- 172.21.15.88:0/260363032 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f923c100110 0x7f923c100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.230+0000 7f9240c5f700 1 --2- 172.21.15.88:0/260363032 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f923c0fed60 0x7f923c0ff1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.230+0000 7f9240c5f700 1 -- 172.21.15.88:0/260363032 >> 172.21.15.88:0/260363032 conn(0x7f923c0f98a0 msgr2=0x7f923c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:27.215 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.230+0000 7f9240c5f700 1 -- 172.21.15.88:0/260363032 shutdown_connections 2023-07-14T17:44:27.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.230+0000 7f9240c5f700 1 -- 172.21.15.88:0/260363032 wait complete. 2023-07-14T17:44:27.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f9240c5f700 1 Processor -- start 2023-07-14T17:44:27.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f9240c5f700 1 -- start start 2023-07-14T17:44:27.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f9240c5f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f923c0fed60 0x7f923c1937a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:27.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f9240c5f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f923c0ff710 0x7f923c193ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:27.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f9240c5f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f923c100110 0x7f923c19ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:27.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f9240c5f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923c19b320 con 0x7f923c0ff710 2023-07-14T17:44:27.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f9240c5f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923c19b460 con 0x7f923c0fed60 2023-07-14T17:44:27.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f9240c5f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923c19b5f0 con 0x7f923c100110 2023-07-14T17:44:27.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f9239d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f923c0ff710 0x7f923c193ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:27.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f9239d9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f923c0ff710 0x7f923c193ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43114/0 (socket says 172.21.15.88:43114) 2023-07-14T17:44:27.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f9239d9b700 1 -- 172.21.15.88:0/1610889711 learned_addr learned my addr 172.21.15.88:0/1610889711 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:27.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f923a59c700 1 --2- 172.21.15.88:0/1610889711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f923c0fed60 0x7f923c1937a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:27.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f923ad9d700 1 --2- 172.21.15.88:0/1610889711 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f923c100110 0x7f923c19ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:27.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f9239d9b700 1 -- 172.21.15.88:0/1610889711 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f923c100110 msgr2=0x7f923c19ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:27.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f9239d9b700 1 --2- 172.21.15.88:0/1610889711 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f923c100110 0x7f923c19ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f9239d9b700 1 -- 172.21.15.88:0/1610889711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f923c0fed60 msgr2=0x7f923c1937a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:27.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f9239d9b700 1 --2- 172.21.15.88:0/1610889711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f923c0fed60 0x7f923c1937a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f9239d9b700 1 -- 172.21.15.88:0/1610889711 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9234009d00 con 0x7f923c0ff710 2023-07-14T17:44:27.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f9239d9b700 1 --2- 172.21.15.88:0/1610889711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f923c0ff710 0x7f923c193ce0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f922c00c920 tx=0x7f922c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:27.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.231+0000 7f922b7fe700 1 -- 172.21.15.88:0/1610889711 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c007b00 con 0x7f923c0ff710 2023-07-14T17:44:27.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.232+0000 7f9240c5f700 1 -- 172.21.15.88:0/1610889711 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f923c19b8d0 con 0x7f923c0ff710 2023-07-14T17:44:27.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.232+0000 7f922b7fe700 1 -- 172.21.15.88:0/1610889711 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f922c00ce70 con 0x7f923c0ff710 2023-07-14T17:44:27.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.232+0000 7f922b7fe700 1 -- 172.21.15.88:0/1610889711 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c0053e0 con 0x7f923c0ff710 2023-07-14T17:44:27.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.232+0000 7f9240c5f700 1 -- 172.21.15.88:0/1610889711 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f923c19be30 con 0x7f923c0ff710 2023-07-14T17:44:27.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.232+0000 7f9240c5f700 1 -- 172.21.15.88:0/1610889711 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f923c105b00 con 0x7f923c0ff710 2023-07-14T17:44:27.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.233+0000 7f922b7fe700 1 -- 172.21.15.88:0/1610889711 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f922c005580 con 0x7f923c0ff710 2023-07-14T17:44:27.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.234+0000 7f922b7fe700 1 --2- 172.21.15.88:0/1610889711 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f92240a3360 0x7f92240a5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:27.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.234+0000 7f922b7fe700 1 -- 172.21.15.88:0/1610889711 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f922c0cc810 con 0x7f923c0ff710 2023-07-14T17:44:27.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.234+0000 7f923a59c700 1 --2- 172.21.15.88:0/1610889711 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f92240a3360 0x7f92240a5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:27.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.235+0000 7f923a59c700 1 --2- 172.21.15.88:0/1610889711 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f92240a3360 0x7f92240a5810 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f923c067580 tx=0x7f923000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:27.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.238+0000 7f922b7fe700 1 -- 172.21.15.88:0/1610889711 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f922c095c50 con 0x7f923c0ff710 2023-07-14T17:44:27.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.426+0000 7f9240c5f700 1 -- 172.21.15.88:0/1610889711 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f923c02cd20 con 0x7f923c0ff710 2023-07-14T17:44:27.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.426+0000 7f922b7fe700 1 -- 172.21.15.88:0/1610889711 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f922c099270 con 0x7f923c0ff710 2023-07-14T17:44:27.409 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-07-14T17:44:27.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.428+0000 7f9240c5f700 1 -- 172.21.15.88:0/1610889711 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f92240a3360 msgr2=0x7f92240a5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:27.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.428+0000 7f9240c5f700 1 --2- 172.21.15.88:0/1610889711 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f92240a3360 0x7f92240a5810 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f923c067580 tx=0x7f923000b410 comp rx=0 tx=0).stop 2023-07-14T17:44:27.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.428+0000 7f9240c5f700 1 -- 172.21.15.88:0/1610889711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f923c0ff710 msgr2=0x7f923c193ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:27.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.428+0000 7f9240c5f700 1 --2- 172.21.15.88:0/1610889711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f923c0ff710 0x7f923c193ce0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f922c00c920 tx=0x7f922c00cc30 comp rx=0 tx=0).stop 2023-07-14T17:44:27.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.428+0000 7f9240c5f700 1 -- 172.21.15.88:0/1610889711 shutdown_connections 2023-07-14T17:44:27.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.429+0000 7f9240c5f700 1 --2- 172.21.15.88:0/1610889711 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f923c100110 0x7f923c19ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.429+0000 7f9240c5f700 1 --2- 172.21.15.88:0/1610889711 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f923c0fed60 0x7f923c1937a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.429+0000 7f9240c5f700 1 --2- 172.21.15.88:0/1610889711 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f92240a3360 0x7f92240a5810 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.429+0000 7f9240c5f700 1 --2- 172.21.15.88:0/1610889711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f923c0ff710 0x7f923c193ce0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.429+0000 7f9240c5f700 1 -- 172.21.15.88:0/1610889711 >> 172.21.15.88:0/1610889711 conn(0x7f923c0f98a0 msgr2=0x7f923c0fbb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:27.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.429+0000 7f9240c5f700 1 -- 172.21.15.88:0/1610889711 shutdown_connections 2023-07-14T17:44:27.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.429+0000 7f9240c5f700 1 -- 172.21.15.88:0/1610889711 wait complete. 2023-07-14T17:44:27.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:27 smithi178 ceph-mon[129368]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 597 B/s wr, 12 op/s 2023-07-14T17:44:27.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:27 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3791849021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:44:27.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:27 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/340866552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:27.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:27 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/4220777174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:27.422 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:27 smithi088 ceph-mon[127465]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 597 B/s wr, 12 op/s 2023-07-14T17:44:27.422 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:27 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3791849021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:44:27.422 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:27 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/340866552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:27.422 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:27 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/4220777174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:27.423 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T17:44:27.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:27 smithi107 ceph-mon[128999]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 597 B/s wr, 12 op/s 2023-07-14T17:44:27.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:27 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3791849021' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:44:27.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:27 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/340866552' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:27.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:27 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/4220777174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:27.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.565+0000 7fd1b49ce700 1 -- 172.21.15.88:0/2291863218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b00ffd00 msgr2=0x7fd1b0100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:27.548 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.565+0000 7fd1b49ce700 1 --2- 172.21.15.88:0/2291863218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b00ffd00 0x7fd1b0100150 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009990 tx=0x7fd1a4009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.565+0000 7fd1b49ce700 1 -- 172.21.15.88:0/2291863218 shutdown_connections 2023-07-14T17:44:27.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.565+0000 7fd1b49ce700 1 --2- 172.21.15.88:0/2291863218 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1b0100690 0x7fd1b0106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.565+0000 7fd1b49ce700 1 --2- 172.21.15.88:0/2291863218 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1b00feb10 0x7fd1b00fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.565+0000 7fd1b49ce700 1 --2- 172.21.15.88:0/2291863218 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b00ffd00 0x7fd1b0100150 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.549 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.565+0000 7fd1b49ce700 1 -- 172.21.15.88:0/2291863218 >> 172.21.15.88:0/2291863218 conn(0x7fd1b00fa130 msgr2=0x7fd1b00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:27.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.566+0000 7fd1b49ce700 1 -- 172.21.15.88:0/2291863218 shutdown_connections 2023-07-14T17:44:27.550 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.566+0000 7fd1b49ce700 1 -- 172.21.15.88:0/2291863218 wait complete. 2023-07-14T17:44:27.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.566+0000 7fd1b49ce700 1 Processor -- start 2023-07-14T17:44:27.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.566+0000 7fd1b49ce700 1 -- start start 2023-07-14T17:44:27.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.566+0000 7fd1b49ce700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1b00feb10 0x7fd1b0069360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:27.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.566+0000 7fd1b49ce700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b00ffd00 0x7fd1b019a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:27.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.566+0000 7fd1b49ce700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1b0100690 0x7fd1b019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:27.552 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.566+0000 7fd1b49ce700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b019d550 con 0x7fd1b00ffd00 2023-07-14T17:44:27.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.566+0000 7fd1b49ce700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b019d6c0 con 0x7fd1b00feb10 2023-07-14T17:44:27.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.566+0000 7fd1b49ce700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b019d880 con 0x7fd1b0100690 2023-07-14T17:44:27.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.567+0000 7fd1add9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b00ffd00 0x7fd1b019a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:27.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.567+0000 7fd1add9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b00ffd00 0x7fd1b019a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43128/0 (socket says 172.21.15.88:43128) 2023-07-14T17:44:27.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.567+0000 7fd1ae59c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1b00feb10 0x7fd1b0069360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:27.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.567+0000 7fd1add9b700 1 -- 172.21.15.88:0/2141083528 learned_addr learned my addr 172.21.15.88:0/2141083528 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:27.553 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.567+0000 7fd1aed9d700 1 --2- 172.21.15.88:0/2141083528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1b0100690 0x7fd1b019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:27.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.567+0000 7fd1add9b700 1 -- 172.21.15.88:0/2141083528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1b0100690 msgr2=0x7fd1b019cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:27.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.567+0000 7fd1add9b700 1 --2- 172.21.15.88:0/2141083528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1b0100690 0x7fd1b019cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.567+0000 7fd1add9b700 1 -- 172.21.15.88:0/2141083528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1b00feb10 msgr2=0x7fd1b0069360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:27.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.567+0000 7fd1add9b700 1 --2- 172.21.15.88:0/2141083528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1b00feb10 0x7fd1b0069360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.554 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.567+0000 7fd1add9b700 1 -- 172.21.15.88:0/2141083528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a40095f0 con 0x7fd1b00ffd00 2023-07-14T17:44:27.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.567+0000 7fd1aed9d700 1 --2- 172.21.15.88:0/2141083528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1b0100690 0x7fd1b019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:27.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.567+0000 7fd1add9b700 1 --2- 172.21.15.88:0/2141083528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b00ffd00 0x7fd1b019a730 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4005910 tx=0x7fd1a40047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:27.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.567+0000 7fd19f7fe700 1 -- 172.21.15.88:0/2141083528 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a401d070 con 0x7fd1b00ffd00 2023-07-14T17:44:27.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.567+0000 7fd1b49ce700 1 -- 172.21.15.88:0/2141083528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b019db00 con 0x7fd1b00ffd00 2023-07-14T17:44:27.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.567+0000 7fd19f7fe700 1 -- 172.21.15.88:0/2141083528 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd1a40041f0 con 0x7fd1b00ffd00 2023-07-14T17:44:27.555 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.567+0000 7fd19f7fe700 1 -- 172.21.15.88:0/2141083528 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4022e90 con 0x7fd1b00ffd00 2023-07-14T17:44:27.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.567+0000 7fd1b49ce700 1 -- 172.21.15.88:0/2141083528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b019df20 con 0x7fd1b00ffd00 2023-07-14T17:44:27.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.568+0000 7fd1b49ce700 1 -- 172.21.15.88:0/2141083528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b010a2f0 con 0x7fd1b00ffd00 2023-07-14T17:44:27.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.569+0000 7fd19f7fe700 1 -- 172.21.15.88:0/2141083528 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd1a4003d60 con 0x7fd1b00ffd00 2023-07-14T17:44:27.556 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.570+0000 7fd19f7fe700 1 --2- 172.21.15.88:0/2141083528 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd1980a1150 0x7fd1980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:27.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.570+0000 7fd1ae59c700 1 --2- 172.21.15.88:0/2141083528 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd1980a1150 0x7fd1980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:27.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.570+0000 7fd19f7fe700 1 -- 172.21.15.88:0/2141083528 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd1a40ce110 con 0x7fd1b00ffd00 2023-07-14T17:44:27.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.570+0000 7fd1ae59c700 1 --2- 172.21.15.88:0/2141083528 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd1980a1150 0x7fd1980a3600 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd1b00ffb60 tx=0x7fd1a0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:27.557 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.574+0000 7fd19f7fe700 1 -- 172.21.15.88:0/2141083528 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd1a405c780 con 0x7fd1b00ffd00 2023-07-14T17:44:27.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.766+0000 7fd1b49ce700 1 -- 172.21.15.88:0/2141083528 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd1b0105b10 con 0x7fd1b00ffd00 2023-07-14T17:44:27.749 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.767+0000 7fd19f7fe700 1 -- 172.21.15.88:0/2141083528 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd1a4097580 con 0x7fd1b00ffd00 2023-07-14T17:44:27.750 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 64 2023-07-14T17:44:27.751 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.769+0000 7fd1b49ce700 1 -- 172.21.15.88:0/2141083528 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd1980a1150 msgr2=0x7fd1980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:27.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.769+0000 7fd1b49ce700 1 --2- 172.21.15.88:0/2141083528 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd1980a1150 0x7fd1980a3600 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd1b00ffb60 tx=0x7fd1a0009190 comp rx=0 tx=0).stop 2023-07-14T17:44:27.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.769+0000 7fd1b49ce700 1 -- 172.21.15.88:0/2141083528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b00ffd00 msgr2=0x7fd1b019a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:27.752 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.769+0000 7fd1b49ce700 1 --2- 172.21.15.88:0/2141083528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b00ffd00 0x7fd1b019a730 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4005910 tx=0x7fd1a40047a0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.769+0000 7fd1b49ce700 1 -- 172.21.15.88:0/2141083528 shutdown_connections 2023-07-14T17:44:27.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.769+0000 7fd1b49ce700 1 --2- 172.21.15.88:0/2141083528 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd1b0100690 0x7fd1b019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.769+0000 7fd1b49ce700 1 --2- 172.21.15.88:0/2141083528 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd1b00feb10 0x7fd1b0069360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.769+0000 7fd1b49ce700 1 --2- 172.21.15.88:0/2141083528 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd1980a1150 0x7fd1980a3600 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.769+0000 7fd1b49ce700 1 --2- 172.21.15.88:0/2141083528 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1b00ffd00 0x7fd1b019a730 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.769+0000 7fd1b49ce700 1 -- 172.21.15.88:0/2141083528 >> 172.21.15.88:0/2141083528 conn(0x7fd1b00fa130 msgr2=0x7fd1b00fc440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:27.754 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.769+0000 7fd1b49ce700 1 -- 172.21.15.88:0/2141083528 shutdown_connections 2023-07-14T17:44:27.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.769+0000 7fd1b49ce700 1 -- 172.21.15.88:0/2141083528 wait complete. 2023-07-14T17:44:27.763 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T17:44:27.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.905+0000 7ff6c553e700 1 -- 172.21.15.88:0/4081217743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c00ffd00 msgr2=0x7ff6c0100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:27.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.905+0000 7ff6c553e700 1 --2- 172.21.15.88:0/4081217743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c00ffd00 0x7ff6c0100150 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0009990 tx=0x7ff6b0009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.905+0000 7ff6c553e700 1 -- 172.21.15.88:0/4081217743 shutdown_connections 2023-07-14T17:44:27.888 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.905+0000 7ff6c553e700 1 --2- 172.21.15.88:0/4081217743 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6c0100690 0x7ff6c0106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.905+0000 7ff6c553e700 1 --2- 172.21.15.88:0/4081217743 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6c00feb10 0x7ff6c00fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.905+0000 7ff6c553e700 1 --2- 172.21.15.88:0/4081217743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c00ffd00 0x7ff6c0100150 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.905+0000 7ff6c553e700 1 -- 172.21.15.88:0/4081217743 >> 172.21.15.88:0/4081217743 conn(0x7ff6c00fa130 msgr2=0x7ff6c00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:27.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.905+0000 7ff6c553e700 1 -- 172.21.15.88:0/4081217743 shutdown_connections 2023-07-14T17:44:27.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.905+0000 7ff6c553e700 1 -- 172.21.15.88:0/4081217743 wait complete. 2023-07-14T17:44:27.889 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.906+0000 7ff6c553e700 1 Processor -- start 2023-07-14T17:44:27.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.906+0000 7ff6c553e700 1 -- start start 2023-07-14T17:44:27.891 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.906+0000 7ff6c553e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6c00feb10 0x7ff6c0068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:27.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.906+0000 7ff6c553e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c00ffd00 0x7ff6c019ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:27.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.906+0000 7ff6c553e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6c0100690 0x7ff6c019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:27.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.906+0000 7ff6c553e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c019f610 con 0x7ff6c00ffd00 2023-07-14T17:44:27.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.906+0000 7ff6c553e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c019f780 con 0x7ff6c0100690 2023-07-14T17:44:27.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.906+0000 7ff6c553e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c019f940 con 0x7ff6c00feb10 2023-07-14T17:44:27.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.906+0000 7ff6be7fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c00ffd00 0x7ff6c019ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:27.892 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.906+0000 7ff6be7fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c00ffd00 0x7ff6c019ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43142/0 (socket says 172.21.15.88:43142) 2023-07-14T17:44:27.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.906+0000 7ff6be7fc700 1 -- 172.21.15.88:0/2917064616 learned_addr learned my addr 172.21.15.88:0/2917064616 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:27.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.906+0000 7ff6bf7fe700 1 --2- 172.21.15.88:0/2917064616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6c0100690 0x7ff6c019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:27.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.906+0000 7ff6beffd700 1 --2- 172.21.15.88:0/2917064616 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6c00feb10 0x7ff6c0068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:27.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.907+0000 7ff6be7fc700 1 -- 172.21.15.88:0/2917064616 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6c00feb10 msgr2=0x7ff6c0068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:27.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.907+0000 7ff6be7fc700 1 --2- 172.21.15.88:0/2917064616 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6c00feb10 0x7ff6c0068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.907+0000 7ff6be7fc700 1 -- 172.21.15.88:0/2917064616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6c0100690 msgr2=0x7ff6c019ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:27.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.907+0000 7ff6be7fc700 1 --2- 172.21.15.88:0/2917064616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6c0100690 0x7ff6c019ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:27.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.907+0000 7ff6be7fc700 1 -- 172.21.15.88:0/2917064616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b00095f0 con 0x7ff6c00ffd00 2023-07-14T17:44:27.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.907+0000 7ff6beffd700 1 --2- 172.21.15.88:0/2917064616 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6c00feb10 0x7ff6c0068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:27.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.907+0000 7ff6be7fc700 1 --2- 172.21.15.88:0/2917064616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c00ffd00 0x7ff6c019ca10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff6b00095c0 tx=0x7ff6b0005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:27.894 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.907+0000 7ff6b7fff700 1 -- 172.21.15.88:0/2917064616 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b001d070 con 0x7ff6c00ffd00 2023-07-14T17:44:27.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.907+0000 7ff6b7fff700 1 -- 172.21.15.88:0/2917064616 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff6b0003fc0 con 0x7ff6c00ffd00 2023-07-14T17:44:27.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.907+0000 7ff6c553e700 1 -- 172.21.15.88:0/2917064616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c019fbc0 con 0x7ff6c00ffd00 2023-07-14T17:44:27.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.907+0000 7ff6b7fff700 1 -- 172.21.15.88:0/2917064616 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0022d00 con 0x7ff6c00ffd00 2023-07-14T17:44:27.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.907+0000 7ff6c553e700 1 -- 172.21.15.88:0/2917064616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c019ffe0 con 0x7ff6c00ffd00 2023-07-14T17:44:27.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.908+0000 7ff6c553e700 1 -- 172.21.15.88:0/2917064616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c010a200 con 0x7ff6c00ffd00 2023-07-14T17:44:27.895 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.909+0000 7ff6b7fff700 1 -- 172.21.15.88:0/2917064616 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff6b0003b30 con 0x7ff6c00ffd00 2023-07-14T17:44:27.896 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.910+0000 7ff6b7fff700 1 --2- 172.21.15.88:0/2917064616 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff6ac0a1150 0x7ff6ac0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:27.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.910+0000 7ff6beffd700 1 --2- 172.21.15.88:0/2917064616 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff6ac0a1150 0x7ff6ac0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:27.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.910+0000 7ff6b7fff700 1 -- 172.21.15.88:0/2917064616 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff6b00ce770 con 0x7ff6c00ffd00 2023-07-14T17:44:27.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.910+0000 7ff6beffd700 1 --2- 172.21.15.88:0/2917064616 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff6ac0a1150 0x7ff6ac0a3600 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8005950 tx=0x7ff6a80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:27.897 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:27.913+0000 7ff6b7fff700 1 -- 172.21.15.88:0/2917064616 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff6b0097c40 con 0x7ff6c00ffd00 2023-07-14T17:44:28.089 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.107+0000 7ff6c553e700 1 -- 172.21.15.88:0/2917064616 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff6c0105ae0 con 0x7ff6c00ffd00 2023-07-14T17:44:28.090 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.108+0000 7ff6b7fff700 1 -- 172.21.15.88:0/2917064616 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff6b0022e60 con 0x7ff6c00ffd00 2023-07-14T17:44:28.090 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-07-14T17:44:28.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.110+0000 7ff6c553e700 1 -- 172.21.15.88:0/2917064616 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff6ac0a1150 msgr2=0x7ff6ac0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.110+0000 7ff6c553e700 1 --2- 172.21.15.88:0/2917064616 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff6ac0a1150 0x7ff6ac0a3600 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8005950 tx=0x7ff6a80058e0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.110+0000 7ff6c553e700 1 -- 172.21.15.88:0/2917064616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c00ffd00 msgr2=0x7ff6c019ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.093 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.110+0000 7ff6c553e700 1 --2- 172.21.15.88:0/2917064616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c00ffd00 0x7ff6c019ca10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff6b00095c0 tx=0x7ff6b0005ef0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.110+0000 7ff6c553e700 1 -- 172.21.15.88:0/2917064616 shutdown_connections 2023-07-14T17:44:28.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.110+0000 7ff6c553e700 1 --2- 172.21.15.88:0/2917064616 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff6c0100690 0x7ff6c019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.094 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.110+0000 7ff6c553e700 1 --2- 172.21.15.88:0/2917064616 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6c00feb10 0x7ff6c0068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.110+0000 7ff6c553e700 1 --2- 172.21.15.88:0/2917064616 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff6ac0a1150 0x7ff6ac0a3600 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.110+0000 7ff6c553e700 1 --2- 172.21.15.88:0/2917064616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6c00ffd00 0x7ff6c019ca10 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.110+0000 7ff6c553e700 1 -- 172.21.15.88:0/2917064616 >> 172.21.15.88:0/2917064616 conn(0x7ff6c00fa130 msgr2=0x7ff6c00fc480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:28.095 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.110+0000 7ff6c553e700 1 -- 172.21.15.88:0/2917064616 shutdown_connections 2023-07-14T17:44:28.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.110+0000 7ff6c553e700 1 -- 172.21.15.88:0/2917064616 wait complete. 2023-07-14T17:44:28.103 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:28 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1501241673' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:44:28.103 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:28 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1610889711' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:44:28.104 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:28 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2141083528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:44:28.105 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:44:28.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.245+0000 7f52f9d6e700 1 -- 172.21.15.88:0/2403699443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52f40fe490 msgr2=0x7f52f40fe8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.228 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.245+0000 7f52f9d6e700 1 --2- 172.21.15.88:0/2403699443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52f40fe490 0x7f52f40fe8a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f52e4009990 tx=0x7f52e4009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.245+0000 7f52f9d6e700 1 -- 172.21.15.88:0/2403699443 shutdown_connections 2023-07-14T17:44:28.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.245+0000 7f52f9d6e700 1 --2- 172.21.15.88:0/2403699443 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52f40ff790 0x7f52f40ffc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.245+0000 7f52f9d6e700 1 --2- 172.21.15.88:0/2403699443 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52f40fe490 0x7f52f40fe8a0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.245+0000 7f52f9d6e700 1 --2- 172.21.15.88:0/2403699443 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f52f40fede0 0x7f52f40ff250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.245+0000 7f52f9d6e700 1 -- 172.21.15.88:0/2403699443 >> 172.21.15.88:0/2403699443 conn(0x7f52f40fa100 msgr2=0x7f52f40fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:28.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.246+0000 7f52f9d6e700 1 -- 172.21.15.88:0/2403699443 shutdown_connections 2023-07-14T17:44:28.229 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.246+0000 7f52f9d6e700 1 -- 172.21.15.88:0/2403699443 wait complete. 2023-07-14T17:44:28.230 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.246+0000 7f52f9d6e700 1 Processor -- start 2023-07-14T17:44:28.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.246+0000 7f52f9d6e700 1 -- start start 2023-07-14T17:44:28.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.246+0000 7f52f9d6e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f52f40fe490 0x7f52f4197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.231 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.246+0000 7f52f9d6e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52f40fede0 0x7f52f4197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.246+0000 7f52f9d6e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52f40ff790 0x7f52f419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.246+0000 7f52f9d6e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f419f5d0 con 0x7f52f40ff790 2023-07-14T17:44:28.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.246+0000 7f52f9d6e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f419f710 con 0x7f52f40fede0 2023-07-14T17:44:28.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.246+0000 7f52f9d6e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f419f8a0 con 0x7f52f40fe490 2023-07-14T17:44:28.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.247+0000 7f52f3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52f40ff790 0x7f52f419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.232 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.247+0000 7f52f3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52f40ff790 0x7f52f419f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43158/0 (socket says 172.21.15.88:43158) 2023-07-14T17:44:28.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.247+0000 7f52f3fff700 1 -- 172.21.15.88:0/2110490728 learned_addr learned my addr 172.21.15.88:0/2110490728 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:28.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.247+0000 7f52f37fe700 1 --2- 172.21.15.88:0/2110490728 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f52f40fe490 0x7f52f4197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.247+0000 7f52f2ffd700 1 --2- 172.21.15.88:0/2110490728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52f40fede0 0x7f52f4197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.247+0000 7f52f3fff700 1 -- 172.21.15.88:0/2110490728 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f52f40fe490 msgr2=0x7f52f4197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.233 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.247+0000 7f52f3fff700 1 --2- 172.21.15.88:0/2110490728 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f52f40fe490 0x7f52f4197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.247+0000 7f52f3fff700 1 -- 172.21.15.88:0/2110490728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52f40fede0 msgr2=0x7f52f4197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.247+0000 7f52f3fff700 1 --2- 172.21.15.88:0/2110490728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52f40fede0 0x7f52f4197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.247+0000 7f52f3fff700 1 -- 172.21.15.88:0/2110490728 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e40095f0 con 0x7f52f40ff790 2023-07-14T17:44:28.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.247+0000 7f52f3fff700 1 --2- 172.21.15.88:0/2110490728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52f40ff790 0x7f52f419f000 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f52ec007ee0 tx=0x7f52ec00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:28.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.247+0000 7f52f0ff9700 1 -- 172.21.15.88:0/2110490728 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec017070 con 0x7f52f40ff790 2023-07-14T17:44:28.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.247+0000 7f52f0ff9700 1 -- 172.21.15.88:0/2110490728 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f52ec00f040 con 0x7f52f40ff790 2023-07-14T17:44:28.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.247+0000 7f52f9d6e700 1 -- 172.21.15.88:0/2110490728 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f419fb80 con 0x7f52f40ff790 2023-07-14T17:44:28.234 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.247+0000 7f52f0ff9700 1 -- 172.21.15.88:0/2110490728 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec005570 con 0x7f52f40ff790 2023-07-14T17:44:28.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.247+0000 7f52f9d6e700 1 -- 172.21.15.88:0/2110490728 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f41a0040 con 0x7f52f40ff790 2023-07-14T17:44:28.235 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.248+0000 7f52f9d6e700 1 -- 172.21.15.88:0/2110490728 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f4109270 con 0x7f52f40ff790 2023-07-14T17:44:28.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.249+0000 7f52f0ff9700 1 -- 172.21.15.88:0/2110490728 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f52ec00db20 con 0x7f52f40ff790 2023-07-14T17:44:28.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.250+0000 7f52f0ff9700 1 --2- 172.21.15.88:0/2110490728 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f52e00a11a0 0x7f52e00a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.250+0000 7f52f0ff9700 1 -- 172.21.15.88:0/2110490728 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f52ec0cc9e0 con 0x7f52f40ff790 2023-07-14T17:44:28.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.250+0000 7f52f37fe700 1 --2- 172.21.15.88:0/2110490728 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f52e00a11a0 0x7f52e00a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.250+0000 7f52f37fe700 1 --2- 172.21.15.88:0/2110490728 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f52e00a11a0 0x7f52e00a3650 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f52e400b5c0 tx=0x7f52e4009f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:28.236 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.253+0000 7f52f0ff9700 1 -- 172.21.15.88:0/2110490728 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f52ec095e20 con 0x7f52f40ff790 2023-07-14T17:44:28.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:28 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1501241673' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:44:28.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:28 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1610889711' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:44:28.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:28 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2141083528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:44:28.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.484+0000 7f52f9d6e700 1 -- 172.21.15.88:0/2110490728 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52f402c8e0 con 0x7f52f40ff790 2023-07-14T17:44:28.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.485+0000 7f52f0ff9700 1 -- 172.21.15.88:0/2110490728 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10885 (secure 0 0 0) 0x7f52ec099440 con 0x7f52f40ff790 2023-07-14T17:44:28.467 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:28.468 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T17:42:42.314442+0000","modified":"2023-07-14T17:43:04.774381+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24473,"mds_2":24469,"mds_3":24454,"mds_4":24449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6839/3046592155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3046592155},{"type":"v1","addr":"172.21.15.88:6839","nonce":3046592155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6841/2509502918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6840","nonce":2509502918},{"type":"v1","addr":"172.21.15.88:6841","nonce":2509502918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6835/2332025068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":2332025068},{"type":"v1","addr":"172.21.15.178:6835","nonce":2332025068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24449":{"gid":24449,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.107:6835/3062637672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3062637672},{"type":"v1","addr":"172.21.15.107:6835","nonce":3062637672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.178:6837/3950404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3950404084},{"type":"v1","addr":"172.21.15.178:6837","nonce":3950404084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.178:6839/2953415077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2953415077},{"type":"v1","addr":"172.21.15.178:6839","nonce":2953415077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/1152608615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1152608615},{"type":"v1","addr":"172.21.15.107:6837","nonce":1152608615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6839/31646548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":31646548},{"type":"v1","addr":"172.21.15.107:6839","nonce":31646548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T17:44:28.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.487+0000 7f52f9d6e700 1 -- 172.21.15.88:0/2110490728 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f52e00a11a0 msgr2=0x7f52e00a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.487+0000 7f52f9d6e700 1 --2- 172.21.15.88:0/2110490728 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f52e00a11a0 0x7f52e00a3650 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f52e400b5c0 tx=0x7f52e4009f00 comp rx=0 tx=0).stop 2023-07-14T17:44:28.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.487+0000 7f52f9d6e700 1 -- 172.21.15.88:0/2110490728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52f40ff790 msgr2=0x7f52f419f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.487+0000 7f52f9d6e700 1 --2- 172.21.15.88:0/2110490728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52f40ff790 0x7f52f419f000 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f52ec007ee0 tx=0x7f52ec00d3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.487+0000 7f52f9d6e700 1 -- 172.21.15.88:0/2110490728 shutdown_connections 2023-07-14T17:44:28.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.487+0000 7f52f9d6e700 1 --2- 172.21.15.88:0/2110490728 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f52e00a11a0 0x7f52e00a3650 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.487+0000 7f52f9d6e700 1 --2- 172.21.15.88:0/2110490728 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52f40ff790 0x7f52f419f000 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.487+0000 7f52f9d6e700 1 --2- 172.21.15.88:0/2110490728 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f52f40fe490 0x7f52f4197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.487+0000 7f52f9d6e700 1 --2- 172.21.15.88:0/2110490728 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f52f40fede0 0x7f52f4197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.487+0000 7f52f9d6e700 1 -- 172.21.15.88:0/2110490728 >> 172.21.15.88:0/2110490728 conn(0x7f52f40fa100 msgr2=0x7f52f40fbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:28.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.487+0000 7f52f9d6e700 1 -- 172.21.15.88:0/2110490728 shutdown_connections 2023-07-14T17:44:28.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.487+0000 7f52f9d6e700 1 -- 172.21.15.88:0/2110490728 wait complete. 2023-07-14T17:44:28.471 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 27 2023-07-14T17:44:28.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:28 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1501241673' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:44:28.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:28 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1610889711' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:44:28.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:28 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2141083528' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:44:28.482 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:44:28.607 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.621+0000 7f9f493eb700 1 -- 172.21.15.88:0/3182533180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f44100100 msgr2=0x7f9f44100510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.621+0000 7f9f493eb700 1 --2- 172.21.15.88:0/3182533180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f44100100 0x7f9f44100510 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9f34009990 tx=0x7f9f34009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.621+0000 7f9f493eb700 1 -- 172.21.15.88:0/3182533180 shutdown_connections 2023-07-14T17:44:28.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.621+0000 7f9f493eb700 1 --2- 172.21.15.88:0/3182533180 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f440ff700 0x7f9f44106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.621+0000 7f9f493eb700 1 --2- 172.21.15.88:0/3182533180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f44100100 0x7f9f44100510 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.621+0000 7f9f493eb700 1 --2- 172.21.15.88:0/3182533180 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f440fed50 0x7f9f440ff1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.621+0000 7f9f493eb700 1 -- 172.21.15.88:0/3182533180 >> 172.21.15.88:0/3182533180 conn(0x7f9f440f98d0 msgr2=0x7f9f440fbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:28.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.621+0000 7f9f493eb700 1 -- 172.21.15.88:0/3182533180 shutdown_connections 2023-07-14T17:44:28.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.621+0000 7f9f493eb700 1 -- 172.21.15.88:0/3182533180 wait complete. 2023-07-14T17:44:28.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.622+0000 7f9f493eb700 1 Processor -- start 2023-07-14T17:44:28.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.622+0000 7f9f493eb700 1 -- start start 2023-07-14T17:44:28.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f493eb700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f440fed50 0x7f9f440fe7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f493eb700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f440ff700 0x7f9f4419a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f493eb700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f44100100 0x7f9f4419cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f493eb700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4419d350 con 0x7f9f440ff700 2023-07-14T17:44:28.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f493eb700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4419d4c0 con 0x7f9f44100100 2023-07-14T17:44:28.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f493eb700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4419d680 con 0x7f9f440fed50 2023-07-14T17:44:28.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f427fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f440ff700 0x7f9f4419a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f427fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f440ff700 0x7f9f4419a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43170/0 (socket says 172.21.15.88:43170) 2023-07-14T17:44:28.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f427fc700 1 -- 172.21.15.88:0/1271041221 learned_addr learned my addr 172.21.15.88:0/1271041221 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:28.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f42ffd700 1 --2- 172.21.15.88:0/1271041221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f440fed50 0x7f9f440fe7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f437fe700 1 --2- 172.21.15.88:0/1271041221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f44100100 0x7f9f4419cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.611 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f427fc700 1 -- 172.21.15.88:0/1271041221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f440fed50 msgr2=0x7f9f440fe7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f427fc700 1 --2- 172.21.15.88:0/1271041221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f440fed50 0x7f9f440fe7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f427fc700 1 -- 172.21.15.88:0/1271041221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f44100100 msgr2=0x7f9f4419cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f427fc700 1 --2- 172.21.15.88:0/1271041221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f44100100 0x7f9f4419cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f427fc700 1 -- 172.21.15.88:0/1271041221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f340095f0 con 0x7f9f440ff700 2023-07-14T17:44:28.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f427fc700 1 --2- 172.21.15.88:0/1271041221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f440ff700 0x7f9f4419a530 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800d950 tx=0x7f9f3800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:28.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f2bfff700 1 -- 172.21.15.88:0/1271041221 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3800f890 con 0x7f9f440ff700 2023-07-14T17:44:28.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f493eb700 1 -- 172.21.15.88:0/1271041221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f4419d960 con 0x7f9f440ff700 2023-07-14T17:44:28.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f2bfff700 1 -- 172.21.15.88:0/1271041221 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9f3800fe80 con 0x7f9f440ff700 2023-07-14T17:44:28.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f2bfff700 1 -- 172.21.15.88:0/1271041221 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3800e3b0 con 0x7f9f440ff700 2023-07-14T17:44:28.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.623+0000 7f9f493eb700 1 -- 172.21.15.88:0/1271041221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f4419de70 con 0x7f9f440ff700 2023-07-14T17:44:28.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.624+0000 7f9f493eb700 1 -- 172.21.15.88:0/1271041221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f44106150 con 0x7f9f440ff700 2023-07-14T17:44:28.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.625+0000 7f9f2bfff700 1 -- 172.21.15.88:0/1271041221 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9f38010460 con 0x7f9f440ff700 2023-07-14T17:44:28.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.626+0000 7f9f2bfff700 1 --2- 172.21.15.88:0/1271041221 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9f2c0a11a0 0x7f9f2c0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.626+0000 7f9f42ffd700 1 --2- 172.21.15.88:0/1271041221 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9f2c0a11a0 0x7f9f2c0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.626+0000 7f9f2bfff700 1 -- 172.21.15.88:0/1271041221 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9f380cc9a0 con 0x7f9f440ff700 2023-07-14T17:44:28.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.627+0000 7f9f42ffd700 1 --2- 172.21.15.88:0/1271041221 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9f2c0a11a0 0x7f9f2c0a3650 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9f34006010 tx=0x7f9f34005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:28.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.630+0000 7f9f2bfff700 1 -- 172.21.15.88:0/1271041221 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9f38095de0 con 0x7f9f440ff700 2023-07-14T17:44:28.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.819+0000 7f9f493eb700 1 -- 172.21.15.88:0/1271041221 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f4402cb30 con 0x7f9f440ff700 2023-07-14T17:44:28.802 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.820+0000 7f9f2bfff700 1 -- 172.21.15.88:0/1271041221 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24512 (secure 0 0 0) 0x7f9f38020090 con 0x7f9f440ff700 2023-07-14T17:44:28.803 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:28.804 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":88,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:43:44.887959+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:44:28.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.822+0000 7f9f493eb700 1 -- 172.21.15.88:0/1271041221 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9f2c0a11a0 msgr2=0x7f9f2c0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.822+0000 7f9f493eb700 1 --2- 172.21.15.88:0/1271041221 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9f2c0a11a0 0x7f9f2c0a3650 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9f34006010 tx=0x7f9f34005960 comp rx=0 tx=0).stop 2023-07-14T17:44:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.822+0000 7f9f493eb700 1 -- 172.21.15.88:0/1271041221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f440ff700 msgr2=0x7f9f4419a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.822+0000 7f9f493eb700 1 --2- 172.21.15.88:0/1271041221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f440ff700 0x7f9f4419a530 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9f3800d950 tx=0x7f9f3800dc60 comp rx=0 tx=0).stop 2023-07-14T17:44:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.822+0000 7f9f493eb700 1 -- 172.21.15.88:0/1271041221 shutdown_connections 2023-07-14T17:44:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.822+0000 7f9f493eb700 1 --2- 172.21.15.88:0/1271041221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f9f44100100 0x7f9f4419cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.822+0000 7f9f493eb700 1 --2- 172.21.15.88:0/1271041221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f9f440fed50 0x7f9f440fe7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.822+0000 7f9f493eb700 1 --2- 172.21.15.88:0/1271041221 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f9f2c0a11a0 0x7f9f2c0a3650 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.822+0000 7f9f493eb700 1 --2- 172.21.15.88:0/1271041221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9f440ff700 0x7f9f4419a530 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.822+0000 7f9f493eb700 1 -- 172.21.15.88:0/1271041221 >> 172.21.15.88:0/1271041221 conn(0x7f9f440f98d0 msgr2=0x7f9f44103290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.823+0000 7f9f493eb700 1 -- 172.21.15.88:0/1271041221 shutdown_connections 2023-07-14T17:44:28.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.823+0000 7f9f493eb700 1 -- 172.21.15.88:0/1271041221 wait complete. 2023-07-14T17:44:28.818 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-07-14T17:42:42.314442+0000', 'modified': '2023-07-14T17:43:04.774381+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24506, 'mds_1': 24473, 'mds_2': 24469, 'mds_3': 24454, 'mds_4': 24449}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14616': {'gid': 14616, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.88:6839/3046592155', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6838', 'nonce': 3046592155}, {'type': 'v1', 'addr': '172.21.15.88:6839', 'nonce': 3046592155}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_14628': {'gid': 14628, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.88:6841/2509502918', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6840', 'nonce': 2509502918}, {'type': 'v1', 'addr': '172.21.15.88:6841', 'nonce': 2509502918}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24415': {'gid': 24415, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.178:6833/1716372869', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6832', 'nonce': 1716372869}, {'type': 'v1', 'addr': '172.21.15.178:6833', 'nonce': 1716372869}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24419': {'gid': 24419, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.107:6833/2946270337', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6832', 'nonce': 2946270337}, {'type': 'v1', 'addr': '172.21.15.107:6833', 'nonce': 2946270337}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.178:6835/2332025068', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6834', 'nonce': 2332025068}, {'type': 'v1', 'addr': '172.21.15.178:6835', 'nonce': 2332025068}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24449': {'gid': 24449, 'name': 'e', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.107:6835/3062637672', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6834', 'nonce': 3062637672}, {'type': 'v1', 'addr': '172.21.15.107:6835', 'nonce': 3062637672}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.178:6837/3950404084', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6836', 'nonce': 3950404084}, {'type': 'v1', 'addr': '172.21.15.178:6837', 'nonce': 3950404084}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24469': {'gid': 24469, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.178:6839/2953415077', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.178:6838', 'nonce': 2953415077}, {'type': 'v1', 'addr': '172.21.15.178:6839', 'nonce': 2953415077}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24473': {'gid': 24473, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.107:6837/1152608615', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6836', 'nonce': 1152608615}, {'type': 'v1', 'addr': '172.21.15.107:6837', 'nonce': 1152608615}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24506': {'gid': 24506, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.107:6839/31646548', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.107:6838', 'nonce': 31646548}, {'type': 'v1', 'addr': '172.21.15.107:6839', 'nonce': 31646548}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-07-14T17:44:28.818 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2023-07-14T17:44:28.819 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-07-14T17:44:28.942 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.958+0000 7f1aaeee0700 1 -- 172.21.15.88:0/1058500701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1aa80ffd30 msgr2=0x7f1aa8100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.958+0000 7f1aaeee0700 1 --2- 172.21.15.88:0/1058500701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1aa80ffd30 0x7f1aa8100180 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1a94009990 tx=0x7f1a94009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.958+0000 7f1aaeee0700 1 -- 172.21.15.88:0/1058500701 shutdown_connections 2023-07-14T17:44:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.958+0000 7f1aaeee0700 1 --2- 172.21.15.88:0/1058500701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1aa81006c0 0x7f1aa8106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.958+0000 7f1aaeee0700 1 --2- 172.21.15.88:0/1058500701 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1aa80feb40 0x7f1aa80fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.958+0000 7f1aaeee0700 1 --2- 172.21.15.88:0/1058500701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1aa80ffd30 0x7f1aa8100180 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.943 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.958+0000 7f1aaeee0700 1 -- 172.21.15.88:0/1058500701 >> 172.21.15.88:0/1058500701 conn(0x7f1aa80fa160 msgr2=0x7f1aa80fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.959+0000 7f1aaeee0700 1 -- 172.21.15.88:0/1058500701 shutdown_connections 2023-07-14T17:44:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.959+0000 7f1aaeee0700 1 -- 172.21.15.88:0/1058500701 wait complete. 2023-07-14T17:44:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.959+0000 7f1aaeee0700 1 Processor -- start 2023-07-14T17:44:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.959+0000 7f1aaeee0700 1 -- start start 2023-07-14T17:44:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.959+0000 7f1aaeee0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1aa80feb40 0x7f1aa8069360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.959+0000 7f1aaeee0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1aa80ffd30 0x7f1aa819a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.959+0000 7f1aaeee0700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1aa81006c0 0x7f1aa819cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.959+0000 7f1aaeee0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa819d3f0 con 0x7f1aa80feb40 2023-07-14T17:44:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.959+0000 7f1aaeee0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa819d560 con 0x7f1aa80ffd30 2023-07-14T17:44:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.959+0000 7f1aaeee0700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa819d720 con 0x7f1aa81006c0 2023-07-14T17:44:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.959+0000 7f1aacc7c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1aa80feb40 0x7f1aa8069360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.959+0000 7f1aacc7c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1aa80feb40 0x7f1aa8069360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43182/0 (socket says 172.21.15.88:43182) 2023-07-14T17:44:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.959+0000 7f1aacc7c700 1 -- 172.21.15.88:0/1303046612 learned_addr learned my addr 172.21.15.88:0/1303046612 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:28.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.960+0000 7f1aa7fff700 1 --2- 172.21.15.88:0/1303046612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1aa80ffd30 0x7f1aa819a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.960+0000 7f1aad47d700 1 --2- 172.21.15.88:0/1303046612 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1aa81006c0 0x7f1aa819cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.960+0000 7f1aacc7c700 1 -- 172.21.15.88:0/1303046612 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1aa81006c0 msgr2=0x7f1aa819cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.960+0000 7f1aacc7c700 1 --2- 172.21.15.88:0/1303046612 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1aa81006c0 0x7f1aa819cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.960+0000 7f1aacc7c700 1 -- 172.21.15.88:0/1303046612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1aa80ffd30 msgr2=0x7f1aa819a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.960+0000 7f1aacc7c700 1 --2- 172.21.15.88:0/1303046612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1aa80ffd30 0x7f1aa819a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.960+0000 7f1aacc7c700 1 -- 172.21.15.88:0/1303046612 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a940095f0 con 0x7f1aa80feb40 2023-07-14T17:44:28.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.960+0000 7f1aad47d700 1 --2- 172.21.15.88:0/1303046612 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1aa81006c0 0x7f1aa819cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:28.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.960+0000 7f1aacc7c700 1 --2- 172.21.15.88:0/1303046612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1aa80feb40 0x7f1aa8069360 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c00d980 tx=0x7f1a9c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:28.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.960+0000 7f1aa5ffb700 1 -- 172.21.15.88:0/1303046612 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c004020 con 0x7f1aa80feb40 2023-07-14T17:44:28.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.960+0000 7f1aaeee0700 1 -- 172.21.15.88:0/1303046612 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa819da00 con 0x7f1aa80feb40 2023-07-14T17:44:28.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.960+0000 7f1aa5ffb700 1 -- 172.21.15.88:0/1303046612 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1a9c009730 con 0x7f1aa80feb40 2023-07-14T17:44:28.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.960+0000 7f1aa5ffb700 1 -- 172.21.15.88:0/1303046612 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c0039c0 con 0x7f1aa80feb40 2023-07-14T17:44:28.947 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.960+0000 7f1aaeee0700 1 -- 172.21.15.88:0/1303046612 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa819df60 con 0x7f1aa80feb40 2023-07-14T17:44:28.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.962+0000 7f1aa5ffb700 1 -- 172.21.15.88:0/1303046612 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1a9c0098a0 con 0x7f1aa80feb40 2023-07-14T17:44:28.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.963+0000 7f1aa5ffb700 1 --2- 172.21.15.88:0/1303046612 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1a980a1150 0x7f1a980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.963+0000 7f1aa7fff700 1 --2- 172.21.15.88:0/1303046612 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1a980a1150 0x7f1a980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.963+0000 7f1aa5ffb700 1 -- 172.21.15.88:0/1303046612 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1a9c0cc950 con 0x7f1aa80feb40 2023-07-14T17:44:28.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.963+0000 7f1aa7fff700 1 --2- 172.21.15.88:0/1303046612 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1a980a1150 0x7f1a980a3600 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1a9400c010 tx=0x7f1a94005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:28.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1abb078b80 1 Processor -- start 2023-07-14T17:44:28.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1abb078b80 1 -- start start 2023-07-14T17:44:28.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1abb078b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef7f10 0x55f23bef8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1abb078b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef8860 0x55f23bef8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1abb078b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef9460 0x55f23bef98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1abb078b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f23bdfcc60 con 0x55f23bef8860 2023-07-14T17:44:28.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1abb078b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f23be4f9f0 con 0x55f23bef7f10 2023-07-14T17:44:28.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1abb078b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f23be48e80 con 0x55f23bef9460 2023-07-14T17:44:28.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1aa7fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef8860 0x55f23bef8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1aa7fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef8860 0x55f23bef8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43192/0 (socket says 172.21.15.88:43192) 2023-07-14T17:44:28.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1aa7fff700 1 -- 172.21.15.88:0/1812679166 learned_addr learned my addr 172.21.15.88:0/1812679166 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:28.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1aacc7c700 1 --2- 172.21.15.88:0/1812679166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef7f10 0x55f23bef8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1aad47d700 1 --2- 172.21.15.88:0/1812679166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef9460 0x55f23bef98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1aa7fff700 1 -- 172.21.15.88:0/1812679166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef9460 msgr2=0x55f23bef98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:28.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1aa7fff700 1 --2- 172.21.15.88:0/1812679166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef9460 0x55f23bef98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1aa7fff700 1 -- 172.21.15.88:0/1812679166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef7f10 msgr2=0x55f23bef8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:28.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1aa7fff700 1 --2- 172.21.15.88:0/1812679166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef7f10 0x55f23bef8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1aa7fff700 1 -- 172.21.15.88:0/1812679166 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f23bc76d90 con 0x55f23bef8860 2023-07-14T17:44:28.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1aa7fff700 1 --2- 172.21.15.88:0/1812679166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef8860 0x55f23bef8e80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1a94015000 tx=0x7f1a94020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13efc93d7772fb9 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:28.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.964+0000 7f1a92ffd700 1 -- 172.21.15.88:0/1812679166 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9402b070 con 0x55f23bef8860 2023-07-14T17:44:28.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.965+0000 7f1a92ffd700 1 -- 172.21.15.88:0/1812679166 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1a9401e040 con 0x55f23bef8860 2023-07-14T17:44:28.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.965+0000 7f1a92ffd700 1 -- 172.21.15.88:0/1812679166 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a94003820 con 0x55f23bef8860 2023-07-14T17:44:28.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.965+0000 7f1abb078b80 1 -- 172.21.15.88:0/1812679166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef8860 msgr2=0x55f23bef8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.965+0000 7f1abb078b80 1 --2- 172.21.15.88:0/1812679166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef8860 0x55f23bef8e80 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1a94015000 tx=0x7f1a94020000 comp rx=0 tx=0).stop 2023-07-14T17:44:28.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.965+0000 7f1abb078b80 1 -- 172.21.15.88:0/1812679166 shutdown_connections 2023-07-14T17:44:28.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.965+0000 7f1abb078b80 1 --2- 172.21.15.88:0/1812679166 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef9460 0x55f23bef98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.965+0000 7f1abb078b80 1 --2- 172.21.15.88:0/1812679166 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef7f10 0x55f23bef8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.965+0000 7f1abb078b80 1 --2- 172.21.15.88:0/1812679166 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef8860 0x55f23bef8e80 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.965+0000 7f1abb078b80 1 -- 172.21.15.88:0/1812679166 >> 172.21.15.88:0/1812679166 conn(0x55f23beed630 msgr2=0x55f23bef2980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:28.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.965+0000 7f1abb078b80 1 -- 172.21.15.88:0/1812679166 shutdown_connections 2023-07-14T17:44:28.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.965+0000 7f1abb078b80 1 -- 172.21.15.88:0/1812679166 wait complete. 2023-07-14T17:44:28.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1abb078b80 1 -- start start 2023-07-14T17:44:28.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1abb078b80 1 Processor -- start 2023-07-14T17:44:28.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1abb078b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef7f10 0x55f23bf01810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1abb078b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef8860 0x55f23bf01d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1abb078b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef9460 0x55f23bf08d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1abb078b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f23bde8a70 con 0x55f23bef9460 2023-07-14T17:44:28.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1abb078b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f23bde8770 con 0x55f23bef8860 2023-07-14T17:44:28.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1abb078b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f23bccc8d0 con 0x55f23bef7f10 2023-07-14T17:44:28.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1abb078b80 10 client.0 _ll_register_callbacks cb 0x55f23bde6710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:28.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1aad47d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef9460 0x55f23bf08d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1aad47d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef9460 0x55f23bf08d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43200/0 (socket says 172.21.15.88:43200) 2023-07-14T17:44:28.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1aad47d700 1 -- 172.21.15.88:0/1114287236 learned_addr learned my addr 172.21.15.88:0/1114287236 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:28.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1aacc7c700 1 --2- 172.21.15.88:0/1114287236 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef7f10 0x55f23bf01810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1aa7fff700 1 --2- 172.21.15.88:0/1114287236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef8860 0x55f23bf01d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1aad47d700 1 -- 172.21.15.88:0/1114287236 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef7f10 msgr2=0x55f23bf01810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:28.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1aad47d700 1 --2- 172.21.15.88:0/1114287236 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef7f10 0x55f23bf01810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1aad47d700 1 -- 172.21.15.88:0/1114287236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef8860 msgr2=0x55f23bf01d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:28.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1aad47d700 1 --2- 172.21.15.88:0/1114287236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef8860 0x55f23bf01d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.966+0000 7f1aad47d700 1 -- 172.21.15.88:0/1114287236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa81078b0 con 0x55f23bef9460 2023-07-14T17:44:28.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.967+0000 7f1aad47d700 1 --2- 172.21.15.88:0/1114287236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef9460 0x55f23bf08d90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0007bf0 tx=0x7f1aa0007f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e566911fcbffc91 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:28.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.967+0000 7f1a87fff700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:28.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.967+0000 7f1a87fff700 1 -- 172.21.15.88:0/1114287236 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa00118d0 con 0x55f23bef9460 2023-07-14T17:44:28.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.967+0000 7f1abb078b80 1 -- 172.21.15.88:0/1114287236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f23bda24a0 con 0x55f23bef9460 2023-07-14T17:44:28.960 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.967+0000 7f1a87fff700 1 -- 172.21.15.88:0/1114287236 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1aa0011a30 con 0x55f23bef9460 2023-07-14T17:44:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.967+0000 7f1a87fff700 1 -- 172.21.15.88:0/1114287236 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1aa0011bf0 con 0x55f23bef9460 2023-07-14T17:44:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.967+0000 7f1abb078b80 10 client.15441 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.967+0000 7f1abb078b80 1 -- 172.21.15.88:0/1114287236 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55f23bc5fc10 con 0x55f23bef9460 2023-07-14T17:44:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.967+0000 7f1a87fff700 1 -- 172.21.15.88:0/1114287236 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55f23bda24a0 con 0x55f23bef9460 2023-07-14T17:44:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.967+0000 7f1a87fff700 1 -- 172.21.15.88:0/1114287236 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1aa0014070 con 0x55f23bef9460 2023-07-14T17:44:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.967+0000 7f1abb078b80 10 client.15441 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.967+0000 7f1abb078b80 10 client.15441 resolve_mds: resolved name 'e' to daemon mds.e 2023-07-14T17:44:28.961 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.967+0000 7f1abb078b80 20 client.15441 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.967+0000 7f1abb078b80 1 --2- 172.21.15.88:0/1114287236 >> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] conn(0x55f23bf0cb90 0x55f23bf0bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.967+0000 7f1abb078b80 4 client.15441 mds_command: new command op to 24449 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T17:44:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.967+0000 7f1abb078b80 1 -- 172.21.15.88:0/1114287236 --> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f23bec6d50 con 0x55f23bf0cb90 2023-07-14T17:44:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.968+0000 7f1aacc7c700 1 --2- 172.21.15.88:0/1114287236 >> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] conn(0x55f23bf0cb90 0x55f23bf0bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.968+0000 7f1a87fff700 10 client.15441 ms_handle_connect on v2:172.21.15.107:6834/3062637672 2023-07-14T17:44:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.968+0000 7f1aacc7c700 1 --2- 172.21.15.88:0/1114287236 >> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] conn(0x55f23bf0cb90 0x55f23bf0bf40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4f24e54dc20248ae server_cookie=fe8bedd52f2ad774 in_seq=0 out_seq=0 2023-07-14T17:44:28.962 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.969+0000 7f1a87fff700 1 -- 172.21.15.88:0/1114287236 <== mds.4 v2:172.21.15.107:6834/3062637672 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55f23bec6d50 con 0x55f23bf0cb90 2023-07-14T17:44:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.969+0000 7f1a87fff700 10 client.15441 handle_command_reply: tid=0 2023-07-14T17:44:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.969+0000 7f1abb078b80 1 client.15441 shutdown 2023-07-14T17:44:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.969+0000 7f1abb078b80 1 -- 172.21.15.88:0/1114287236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef9460 msgr2=0x55f23bf08d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.969+0000 7f1abb078b80 1 --2- 172.21.15.88:0/1114287236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef9460 0x55f23bf08d90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0007bf0 tx=0x7f1aa0007f00 comp rx=0 tx=0).stop 2023-07-14T17:44:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.969+0000 7f1abb078b80 1 -- 172.21.15.88:0/1114287236 shutdown_connections 2023-07-14T17:44:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.969+0000 7f1abb078b80 1 --2- 172.21.15.88:0/1114287236 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef9460 0x55f23bf08d90 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.963 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.969+0000 7f1abb078b80 1 --2- 172.21.15.88:0/1114287236 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef7f10 0x55f23bf01810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.969+0000 7f1abb078b80 1 --2- 172.21.15.88:0/1114287236 >> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] conn(0x55f23bf0cb90 0x55f23bf0bf40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.969+0000 7f1abb078b80 1 --2- 172.21.15.88:0/1114287236 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef8860 0x55f23bf01d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.969+0000 7f1a87fff700 0 client.15441 ms_handle_reset on v2:172.21.15.107:6834/3062637672 2023-07-14T17:44:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.969+0000 7f1abb078b80 1 -- 172.21.15.88:0/1114287236 >> 172.21.15.88:0/1114287236 conn(0x55f23beed630 msgr2=0x55f23bef1600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.969+0000 7f1abb078b80 1 -- 172.21.15.88:0/1114287236 shutdown_connections 2023-07-14T17:44:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.969+0000 7f1abb078b80 1 -- 172.21.15.88:0/1114287236 wait complete. 2023-07-14T17:44:28.964 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.969+0000 7f1abb078b80 20 client.15441 trim_cache size 0 max 16384 2023-07-14T17:44:28.969 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.986+0000 7f1abb078b80 1 Processor -- start 2023-07-14T17:44:28.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.986+0000 7f1abb078b80 1 -- start start 2023-07-14T17:44:28.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1abb078b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef7f10 0x55f23beff7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1abb078b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef8860 0x55f23bf158e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1abb078b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef9460 0x55f23bf0bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1abb078b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f23bcd8de0 con 0x55f23bef9460 2023-07-14T17:44:28.970 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1abb078b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f23be07480 con 0x55f23bef7f10 2023-07-14T17:44:28.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1abb078b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f23bdf2470 con 0x55f23bef8860 2023-07-14T17:44:28.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1aad47d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef9460 0x55f23bf0bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1aad47d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef9460 0x55f23bf0bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43208/0 (socket says 172.21.15.88:43208) 2023-07-14T17:44:28.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1aad47d700 1 -- 172.21.15.88:0/3096587876 learned_addr learned my addr 172.21.15.88:0/3096587876 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:28.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1aacc7c700 1 --2- 172.21.15.88:0/3096587876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef7f10 0x55f23beff7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1aa7fff700 1 --2- 172.21.15.88:0/3096587876 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef8860 0x55f23bf158e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.971 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1aad47d700 1 -- 172.21.15.88:0/3096587876 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef8860 msgr2=0x55f23bf158e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:28.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1aad47d700 1 --2- 172.21.15.88:0/3096587876 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef8860 0x55f23bf158e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1aad47d700 1 -- 172.21.15.88:0/3096587876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef7f10 msgr2=0x55f23beff7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:28.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1aad47d700 1 --2- 172.21.15.88:0/3096587876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef7f10 0x55f23beff7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1aad47d700 1 -- 172.21.15.88:0/3096587876 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f23bc9d200 con 0x55f23bef9460 2023-07-14T17:44:28.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1aad47d700 1 --2- 172.21.15.88:0/3096587876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef9460 0x55f23bf0bf40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0000f80 tx=0x7f1aa00098c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d13e791977d4a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:28.972 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1a937fe700 1 -- 172.21.15.88:0/3096587876 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0004ae0 con 0x55f23bef9460 2023-07-14T17:44:28.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1a937fe700 1 -- 172.21.15.88:0/3096587876 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1aa000f050 con 0x55f23bef9460 2023-07-14T17:44:28.973 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.987+0000 7f1a937fe700 1 -- 172.21.15.88:0/3096587876 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0015b20 con 0x55f23bef9460 2023-07-14T17:44:28.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.988+0000 7f1abb078b80 1 -- 172.21.15.88:0/3096587876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef9460 msgr2=0x55f23bf0bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.988+0000 7f1abb078b80 1 --2- 172.21.15.88:0/3096587876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef9460 0x55f23bf0bf40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0000f80 tx=0x7f1aa00098c0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.988+0000 7f1abb078b80 1 -- 172.21.15.88:0/3096587876 shutdown_connections 2023-07-14T17:44:28.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.988+0000 7f1abb078b80 1 --2- 172.21.15.88:0/3096587876 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef9460 0x55f23bf0bf40 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.976 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.988+0000 7f1abb078b80 1 --2- 172.21.15.88:0/3096587876 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef7f10 0x55f23beff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.988+0000 7f1abb078b80 1 --2- 172.21.15.88:0/3096587876 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef8860 0x55f23bf158e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.988+0000 7f1abb078b80 1 -- 172.21.15.88:0/3096587876 >> 172.21.15.88:0/3096587876 conn(0x55f23beed630 msgr2=0x55f23beeeb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:28.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.988+0000 7f1abb078b80 1 -- 172.21.15.88:0/3096587876 shutdown_connections 2023-07-14T17:44:28.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.988+0000 7f1abb078b80 1 -- 172.21.15.88:0/3096587876 wait complete. 2023-07-14T17:44:28.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.988+0000 7f1abb078b80 1 -- start start 2023-07-14T17:44:28.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.988+0000 7f1abb078b80 1 Processor -- start 2023-07-14T17:44:28.977 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1abb078b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef7f10 0x55f23bef2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1abb078b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef8860 0x55f23bf158e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1abb078b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef9460 0x55f23bf608e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1abb078b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f23bf61020 con 0x55f23bef7f10 2023-07-14T17:44:28.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1abb078b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f23bf61190 con 0x55f23bef9460 2023-07-14T17:44:28.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1abb078b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f23bf61350 con 0x55f23bef8860 2023-07-14T17:44:28.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1abb078b80 10 client.0 _ll_register_callbacks cb 0x7f1aa0009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:28.978 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1aacc7c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef7f10 0x55f23bef2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1aacc7c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef7f10 0x55f23bef2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43210/0 (socket says 172.21.15.88:43210) 2023-07-14T17:44:28.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1aacc7c700 1 -- 172.21.15.88:0/1972642102 learned_addr learned my addr 172.21.15.88:0/1972642102 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:28.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1aa7fff700 1 --2- 172.21.15.88:0/1972642102 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef8860 0x55f23bf158e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1aad47d700 1 --2- 172.21.15.88:0/1972642102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef9460 0x55f23bf608e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1aacc7c700 1 -- 172.21.15.88:0/1972642102 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef8860 msgr2=0x55f23bf158e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:28.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1aacc7c700 1 --2- 172.21.15.88:0/1972642102 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef8860 0x55f23bf158e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1aacc7c700 1 -- 172.21.15.88:0/1972642102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef9460 msgr2=0x55f23bf608e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:28.979 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1aacc7c700 1 --2- 172.21.15.88:0/1972642102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef9460 0x55f23bf608e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1aacc7c700 1 -- 172.21.15.88:0/1972642102 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f23bc5fc10 con 0x55f23bef7f10 2023-07-14T17:44:28.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1aad47d700 1 --2- 172.21.15.88:0/1972642102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef9460 0x55f23bf608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:28.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1aacc7c700 1 --2- 172.21.15.88:0/1972642102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef7f10 0x55f23bef2b20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c096460 tx=0x7f1a9c0999f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30be73ac8c3f6aa9 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:28.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1a87fff700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:28.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1a87fff700 1 -- 172.21.15.88:0/1972642102 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c0a1e40 con 0x55f23bef7f10 2023-07-14T17:44:28.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1a87fff700 1 -- 172.21.15.88:0/1972642102 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1a9c0b20d0 con 0x55f23bef7f10 2023-07-14T17:44:28.980 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.989+0000 7f1abb078b80 1 -- 172.21.15.88:0/1972642102 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1a9401e040 con 0x55f23bef7f10 2023-07-14T17:44:28.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.990+0000 7f1a87fff700 1 -- 172.21.15.88:0/1972642102 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a9c098ee0 con 0x55f23bef7f10 2023-07-14T17:44:28.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.990+0000 7f1abb078b80 10 client.15447 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:28.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.990+0000 7f1abb078b80 1 -- 172.21.15.88:0/1972642102 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55f23bc75260 con 0x55f23bef7f10 2023-07-14T17:44:28.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.990+0000 7f1a87fff700 1 -- 172.21.15.88:0/1972642102 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1a9401e040 con 0x55f23bef7f10 2023-07-14T17:44:28.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.990+0000 7f1a87fff700 1 -- 172.21.15.88:0/1972642102 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1a9c0b71b0 con 0x55f23bef7f10 2023-07-14T17:44:28.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.990+0000 7f1abb078b80 10 client.15447 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:28.981 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.990+0000 7f1abb078b80 10 client.15447 resolve_mds: resolved name 'e' to daemon mds.e 2023-07-14T17:44:28.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.990+0000 7f1abb078b80 20 client.15447 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:28.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.990+0000 7f1abb078b80 1 --2- 172.21.15.88:0/1972642102 >> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] conn(0x55f23bf66120 0x55f23bf62a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:28.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.990+0000 7f1abb078b80 4 client.15447 mds_command: new command op to 24449 tid=0[{"prefix": "status"}] 2023-07-14T17:44:28.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.990+0000 7f1abb078b80 1 -- 172.21.15.88:0/1972642102 --> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f23be41e60 con 0x55f23bf66120 2023-07-14T17:44:28.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.990+0000 7f1aa7fff700 1 --2- 172.21.15.88:0/1972642102 >> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] conn(0x55f23bf66120 0x55f23bf62a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:28.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.991+0000 7f1a87fff700 10 client.15447 ms_handle_connect on v2:172.21.15.107:6834/3062637672 2023-07-14T17:44:28.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.991+0000 7f1aa7fff700 1 --2- 172.21.15.88:0/1972642102 >> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] conn(0x55f23bf66120 0x55f23bf62a10 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=55bb81b13b92b059 server_cookie=449e8b1076fef141 in_seq=0 out_seq=0 2023-07-14T17:44:28.982 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.991+0000 7f1a87fff700 1 -- 172.21.15.88:0/1972642102 <== mds.4 v2:172.21.15.107:6834/3062637672 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55f23be41e60 con 0x55f23bf66120 2023-07-14T17:44:28.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.991+0000 7f1a87fff700 10 client.15447 handle_command_reply: tid=0 2023-07-14T17:44:28.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.991+0000 7f1abb078b80 1 client.15447 shutdown 2023-07-14T17:44:28.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.991+0000 7f1abb078b80 1 -- 172.21.15.88:0/1972642102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef7f10 msgr2=0x55f23bef2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.991+0000 7f1abb078b80 1 --2- 172.21.15.88:0/1972642102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef7f10 0x55f23bef2b20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c096460 tx=0x7f1a9c0999f0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.991+0000 7f1abb078b80 1 -- 172.21.15.88:0/1972642102 shutdown_connections 2023-07-14T17:44:28.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.991+0000 7f1abb078b80 1 --2- 172.21.15.88:0/1972642102 >> [v2:172.21.15.107:6834/3062637672,v1:172.21.15.107:6835/3062637672] conn(0x55f23bf66120 0x55f23bf62a10 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.983 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.991+0000 7f1abb078b80 1 --2- 172.21.15.88:0/1972642102 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55f23bef9460 0x55f23bf608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.991+0000 7f1a87fff700 0 client.15447 ms_handle_reset on v2:172.21.15.107:6834/3062637672 2023-07-14T17:44:28.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.991+0000 7f1abb078b80 1 --2- 172.21.15.88:0/1972642102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55f23bef7f10 0x55f23bef2b20 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.991+0000 7f1abb078b80 1 --2- 172.21.15.88:0/1972642102 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55f23bef8860 0x55f23bf158e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.991+0000 7f1abb078b80 1 -- 172.21.15.88:0/1972642102 >> 172.21.15.88:0/1972642102 conn(0x55f23beed630 msgr2=0x55f23beff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:28.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.991+0000 7f1abb078b80 1 -- 172.21.15.88:0/1972642102 shutdown_connections 2023-07-14T17:44:28.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.991+0000 7f1abb078b80 1 -- 172.21.15.88:0/1972642102 wait complete. 2023-07-14T17:44:28.984 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.991+0000 7f1abb078b80 20 client.15447 trim_cache size 0 max 16384 2023-07-14T17:44:28.985 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-07-14T17:44:28.985 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "b462a556-226c-11ee-9b33-001a4aab830c", 2023-07-14T17:44:28.985 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 4, 2023-07-14T17:44:28.985 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 24449, 2023-07-14T17:44:28.985 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2023-07-14T17:44:28.986 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2023-07-14T17:44:28.986 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2023-07-14T17:44:28.986 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 85.231065525000005, 2023-07-14T17:44:28.986 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 27, 2023-07-14T17:44:28.986 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 88, 2023-07-14T17:44:28.986 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 86, 2023-07-14T17:44:28.986 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 127.227514372 2023-07-14T17:44:28.987 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-07-14T17:44:28.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.992+0000 7f1aaeee0700 1 -- 172.21.15.88:0/1303046612 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1a980a1150 msgr2=0x7f1a980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.992+0000 7f1aaeee0700 1 --2- 172.21.15.88:0/1303046612 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1a980a1150 0x7f1a980a3600 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1a9400c010 tx=0x7f1a94005fb0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.992+0000 7f1aaeee0700 1 -- 172.21.15.88:0/1303046612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1aa80feb40 msgr2=0x7f1aa8069360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:28.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.992+0000 7f1aaeee0700 1 --2- 172.21.15.88:0/1303046612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1aa80feb40 0x7f1aa8069360 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c00d980 tx=0x7f1a9c00dc90 comp rx=0 tx=0).stop 2023-07-14T17:44:28.988 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.992+0000 7f1aaeee0700 1 -- 172.21.15.88:0/1303046612 shutdown_connections 2023-07-14T17:44:28.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.992+0000 7f1aaeee0700 1 --2- 172.21.15.88:0/1303046612 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1aa81006c0 0x7f1aa819cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.992+0000 7f1aaeee0700 1 --2- 172.21.15.88:0/1303046612 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1a980a1150 0x7f1a980a3600 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.992+0000 7f1aaeee0700 1 --2- 172.21.15.88:0/1303046612 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1aa80feb40 0x7f1aa8069360 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.992+0000 7f1aaeee0700 1 --2- 172.21.15.88:0/1303046612 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1aa80ffd30 0x7f1aa819a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:28.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.992+0000 7f1aaeee0700 1 -- 172.21.15.88:0/1303046612 >> 172.21.15.88:0/1303046612 conn(0x7f1aa80fa160 msgr2=0x7f1aa80fc480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:28.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.992+0000 7f1aaeee0700 1 -- 172.21.15.88:0/1303046612 shutdown_connections 2023-07-14T17:44:28.989 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:28.992+0000 7f1aaeee0700 1 -- 172.21.15.88:0/1303046612 wait complete. 2023-07-14T17:44:28.990 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-07-14T17:44:29.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.129+0000 7f53660e2700 1 -- 172.21.15.88:0/743782794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53600ffd00 msgr2=0x7f5360100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.112 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.129+0000 7f53660e2700 1 --2- 172.21.15.88:0/743782794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53600ffd00 0x7f5360100150 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f5354009990 tx=0x7f5354009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.129+0000 7f53660e2700 1 -- 172.21.15.88:0/743782794 shutdown_connections 2023-07-14T17:44:29.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.129+0000 7f53660e2700 1 --2- 172.21.15.88:0/743782794 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f5360100690 0x7f5360106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.129+0000 7f53660e2700 1 --2- 172.21.15.88:0/743782794 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f53600feb10 0x7f53600fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.129+0000 7f53660e2700 1 --2- 172.21.15.88:0/743782794 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53600ffd00 0x7f5360100150 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.129+0000 7f53660e2700 1 -- 172.21.15.88:0/743782794 >> 172.21.15.88:0/743782794 conn(0x7f53600fa130 msgr2=0x7f53600fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.130+0000 7f53660e2700 1 -- 172.21.15.88:0/743782794 shutdown_connections 2023-07-14T17:44:29.113 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.130+0000 7f53660e2700 1 -- 172.21.15.88:0/743782794 wait complete. 2023-07-14T17:44:29.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.130+0000 7f53660e2700 1 Processor -- start 2023-07-14T17:44:29.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.130+0000 7f53660e2700 1 -- start start 2023-07-14T17:44:29.115 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.130+0000 7f53660e2700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53600feb10 0x7f5360197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.130+0000 7f53660e2700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53600ffd00 0x7f5360198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.130+0000 7f53660e2700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5360100690 0x7f536019f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.130+0000 7f53660e2700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536019f650 con 0x7f53600feb10 2023-07-14T17:44:29.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.130+0000 7f53660e2700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536019f790 con 0x7f53600ffd00 2023-07-14T17:44:29.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.130+0000 7f53660e2700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536019f920 con 0x7f5360100690 2023-07-14T17:44:29.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.131+0000 7f535f7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53600feb10 0x7f5360197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.116 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.131+0000 7f535f7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53600feb10 0x7f5360197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43224/0 (socket says 172.21.15.88:43224) 2023-07-14T17:44:29.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.131+0000 7f535f7fe700 1 -- 172.21.15.88:0/793306660 learned_addr learned my addr 172.21.15.88:0/793306660 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.131+0000 7f535ffff700 1 --2- 172.21.15.88:0/793306660 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5360100690 0x7f536019f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.131+0000 7f535effd700 1 --2- 172.21.15.88:0/793306660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53600ffd00 0x7f5360198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.131+0000 7f535f7fe700 1 -- 172.21.15.88:0/793306660 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5360100690 msgr2=0x7f536019f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.131+0000 7f535f7fe700 1 --2- 172.21.15.88:0/793306660 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5360100690 0x7f536019f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.131+0000 7f535f7fe700 1 -- 172.21.15.88:0/793306660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53600ffd00 msgr2=0x7f5360198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.117 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.131+0000 7f535f7fe700 1 --2- 172.21.15.88:0/793306660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53600ffd00 0x7f5360198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.131+0000 7f535f7fe700 1 -- 172.21.15.88:0/793306660 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53540095f0 con 0x7f53600feb10 2023-07-14T17:44:29.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.131+0000 7f535f7fe700 1 --2- 172.21.15.88:0/793306660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53600feb10 0x7f5360197ad0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f535000c8f0 tx=0x7f535000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.131+0000 7f535cff9700 1 -- 172.21.15.88:0/793306660 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5350004220 con 0x7f53600feb10 2023-07-14T17:44:29.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.131+0000 7f535cff9700 1 -- 172.21.15.88:0/793306660 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5350007d90 con 0x7f53600feb10 2023-07-14T17:44:29.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.131+0000 7f53660e2700 1 -- 172.21.15.88:0/793306660 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536019fc00 con 0x7f53600feb10 2023-07-14T17:44:29.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.131+0000 7f535cff9700 1 -- 172.21.15.88:0/793306660 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5350003be0 con 0x7f53600feb10 2023-07-14T17:44:29.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.131+0000 7f53660e2700 1 -- 172.21.15.88:0/793306660 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53601a0110 con 0x7f53600feb10 2023-07-14T17:44:29.118 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.133+0000 7f535cff9700 1 -- 172.21.15.88:0/793306660 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5350007900 con 0x7f53600feb10 2023-07-14T17:44:29.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.134+0000 7f535cff9700 1 --2- 172.21.15.88:0/793306660 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f53480a11d0 0x7f53480a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.134+0000 7f535effd700 1 --2- 172.21.15.88:0/793306660 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f53480a11d0 0x7f53480a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.134+0000 7f535cff9700 1 -- 172.21.15.88:0/793306660 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f53500cc840 con 0x7f53600feb10 2023-07-14T17:44:29.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.134+0000 7f535effd700 1 --2- 172.21.15.88:0/793306660 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f53480a11d0 0x7f53480a3680 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f535400c010 tx=0x7f535400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f537227ab80 1 Processor -- start 2023-07-14T17:44:29.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f537227ab80 1 -- start start 2023-07-14T17:44:29.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f537227ab80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe7f10 0x55fec8fe8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f537227ab80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe8860 0x55fec8fe8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f537227ab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 0x55fec8fe98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f537227ab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fec8eecc60 con 0x55fec8fe9460 2023-07-14T17:44:29.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f537227ab80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fec8f3f9f0 con 0x55fec8fe7f10 2023-07-14T17:44:29.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f537227ab80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fec8f38e80 con 0x55fec8fe8860 2023-07-14T17:44:29.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f535ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 0x55fec8fe98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.125 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f535ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 0x55fec8fe98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43236/0 (socket says 172.21.15.88:43236) 2023-07-14T17:44:29.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f535ffff700 1 -- 172.21.15.88:0/3233334940 learned_addr learned my addr 172.21.15.88:0/3233334940 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f535effd700 1 --2- 172.21.15.88:0/3233334940 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe8860 0x55fec8fe8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f535ffff700 1 -- 172.21.15.88:0/3233334940 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe8860 msgr2=0x55fec8fe8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f535ffff700 1 --2- 172.21.15.88:0/3233334940 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe8860 0x55fec8fe8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f535ffff700 1 -- 172.21.15.88:0/3233334940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe7f10 msgr2=0x55fec8fe8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f535f7fe700 1 --2- 172.21.15.88:0/3233334940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe7f10 0x55fec8fe8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f535ffff700 1 --2- 172.21.15.88:0/3233334940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe7f10 0x55fec8fe8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f535ffff700 1 -- 172.21.15.88:0/3233334940 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fec8d66d90 con 0x55fec8fe9460 2023-07-14T17:44:29.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.135+0000 7f535f7fe700 1 --2- 172.21.15.88:0/3233334940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe7f10 0x55fec8fe8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:29.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.136+0000 7f535ffff700 1 --2- 172.21.15.88:0/3233334940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 0x55fec8fe98d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5358009dc0 tx=0x7f53580077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f20452712b73720c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.136+0000 7f5345ffb700 1 -- 172.21.15.88:0/3233334940 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358014610 con 0x55fec8fe9460 2023-07-14T17:44:29.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.136+0000 7f5345ffb700 1 -- 172.21.15.88:0/3233334940 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f535800f050 con 0x55fec8fe9460 2023-07-14T17:44:29.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.136+0000 7f5345ffb700 1 -- 172.21.15.88:0/3233334940 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358013870 con 0x55fec8fe9460 2023-07-14T17:44:29.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.136+0000 7f537227ab80 1 -- 172.21.15.88:0/3233334940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 msgr2=0x55fec8fe98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.127 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.136+0000 7f537227ab80 1 --2- 172.21.15.88:0/3233334940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 0x55fec8fe98d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5358009dc0 tx=0x7f53580077e0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.136+0000 7f537227ab80 1 -- 172.21.15.88:0/3233334940 shutdown_connections 2023-07-14T17:44:29.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.136+0000 7f537227ab80 1 --2- 172.21.15.88:0/3233334940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 0x55fec8fe98d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.136+0000 7f537227ab80 1 --2- 172.21.15.88:0/3233334940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe7f10 0x55fec8fe8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.136+0000 7f537227ab80 1 --2- 172.21.15.88:0/3233334940 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe8860 0x55fec8fe8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.136+0000 7f537227ab80 1 -- 172.21.15.88:0/3233334940 >> 172.21.15.88:0/3233334940 conn(0x55fec8fdd630 msgr2=0x55fec8fe2980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.136+0000 7f537227ab80 1 -- 172.21.15.88:0/3233334940 shutdown_connections 2023-07-14T17:44:29.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.136+0000 7f537227ab80 1 -- 172.21.15.88:0/3233334940 wait complete. 2023-07-14T17:44:29.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.137+0000 7f537227ab80 1 -- start start 2023-07-14T17:44:29.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.137+0000 7f537227ab80 1 Processor -- start 2023-07-14T17:44:29.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.137+0000 7f537227ab80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe7f10 0x55fec8ff1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.137+0000 7f537227ab80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe8860 0x55fec8ff1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.137+0000 7f537227ab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 0x55fec8ff8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.137+0000 7f537227ab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fec8ed8a70 con 0x55fec8fe9460 2023-07-14T17:44:29.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.137+0000 7f537227ab80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fec8ed8770 con 0x55fec8fe8860 2023-07-14T17:44:29.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.137+0000 7f537227ab80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fec8dbc8d0 con 0x55fec8fe7f10 2023-07-14T17:44:29.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.137+0000 7f537227ab80 10 client.0 _ll_register_callbacks cb 0x55fec8ed6710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:29.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.137+0000 7f535ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 0x55fec8ff8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.137+0000 7f535ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 0x55fec8ff8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43242/0 (socket says 172.21.15.88:43242) 2023-07-14T17:44:29.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.137+0000 7f535ffff700 1 -- 172.21.15.88:0/1699733063 learned_addr learned my addr 172.21.15.88:0/1699733063 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.137+0000 7f535f7fe700 1 --2- 172.21.15.88:0/1699733063 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe7f10 0x55fec8ff1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.137+0000 7f535effd700 1 --2- 172.21.15.88:0/1699733063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe8860 0x55fec8ff1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f535ffff700 1 -- 172.21.15.88:0/1699733063 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe7f10 msgr2=0x55fec8ff1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f535ffff700 1 --2- 172.21.15.88:0/1699733063 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe7f10 0x55fec8ff1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f535ffff700 1 -- 172.21.15.88:0/1699733063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe8860 msgr2=0x55fec8ff1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f535ffff700 1 --2- 172.21.15.88:0/1699733063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe8860 0x55fec8ff1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f535ffff700 1 -- 172.21.15.88:0/1699733063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5358009a20 con 0x55fec8fe9460 2023-07-14T17:44:29.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f535ffff700 1 --2- 172.21.15.88:0/1699733063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 0x55fec8ff8e90 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f53580090b0 tx=0x7f5358014ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b5e3d7aa2b41448 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f533effd700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:29.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f533effd700 1 -- 172.21.15.88:0/1699733063 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358006000 con 0x55fec8fe9460 2023-07-14T17:44:29.131 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f533effd700 1 -- 172.21.15.88:0/1699733063 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f53580135b0 con 0x55fec8fe9460 2023-07-14T17:44:29.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f537227ab80 1 -- 172.21.15.88:0/1699733063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fec8e924a0 con 0x55fec8fe9460 2023-07-14T17:44:29.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f533effd700 1 -- 172.21.15.88:0/1699733063 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5358013770 con 0x55fec8fe9460 2023-07-14T17:44:29.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f537227ab80 10 client.15459 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:29.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f537227ab80 1 -- 172.21.15.88:0/1699733063 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55fec8d4fc10 con 0x55fec8fe9460 2023-07-14T17:44:29.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f533effd700 1 -- 172.21.15.88:0/1699733063 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55fec8e924a0 con 0x55fec8fe9460 2023-07-14T17:44:29.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f533effd700 1 -- 172.21.15.88:0/1699733063 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5358026990 con 0x55fec8fe9460 2023-07-14T17:44:29.132 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f537227ab80 10 client.15459 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:29.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f537227ab80 10 client.15459 resolve_mds: resolved name 'c' to daemon mds.c 2023-07-14T17:44:29.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f537227ab80 20 client.15459 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:29.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f537227ab80 1 --2- 172.21.15.88:0/1699733063 >> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] conn(0x55fec8ffca70 0x55fec8ffbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f537227ab80 4 client.15459 mds_command: new command op to 24454 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T17:44:29.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.138+0000 7f537227ab80 1 -- 172.21.15.88:0/1699733063 --> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fec8fb6d50 con 0x55fec8ffca70 2023-07-14T17:44:29.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.139+0000 7f535f7fe700 1 --2- 172.21.15.88:0/1699733063 >> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] conn(0x55fec8ffca70 0x55fec8ffbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.139+0000 7f533effd700 10 client.15459 ms_handle_connect on v2:172.21.15.178:6836/3950404084 2023-07-14T17:44:29.133 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.139+0000 7f535f7fe700 1 --2- 172.21.15.88:0/1699733063 >> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] conn(0x55fec8ffca70 0x55fec8ffbea0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=778cae290bcb15b9 server_cookie=9bd120740e1128e7 in_seq=0 out_seq=0 2023-07-14T17:44:29.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.140+0000 7f533effd700 1 -- 172.21.15.88:0/1699733063 <== mds.3 v2:172.21.15.178:6836/3950404084 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55fec8fb6d50 con 0x55fec8ffca70 2023-07-14T17:44:29.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.140+0000 7f533effd700 10 client.15459 handle_command_reply: tid=0 2023-07-14T17:44:29.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.140+0000 7f537227ab80 1 client.15459 shutdown 2023-07-14T17:44:29.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.140+0000 7f537227ab80 1 -- 172.21.15.88:0/1699733063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 msgr2=0x55fec8ff8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.140+0000 7f537227ab80 1 --2- 172.21.15.88:0/1699733063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 0x55fec8ff8e90 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f53580090b0 tx=0x7f5358014ba0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.140+0000 7f537227ab80 1 -- 172.21.15.88:0/1699733063 shutdown_connections 2023-07-14T17:44:29.134 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.140+0000 7f537227ab80 1 --2- 172.21.15.88:0/1699733063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 0x55fec8ff8e90 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.140+0000 7f537227ab80 1 --2- 172.21.15.88:0/1699733063 >> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] conn(0x55fec8ffca70 0x55fec8ffbea0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.140+0000 7f537227ab80 1 --2- 172.21.15.88:0/1699733063 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe7f10 0x55fec8ff1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.140+0000 7f537227ab80 1 --2- 172.21.15.88:0/1699733063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe8860 0x55fec8ff1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.140+0000 7f533effd700 0 client.15459 ms_handle_reset on v2:172.21.15.178:6836/3950404084 2023-07-14T17:44:29.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.140+0000 7f537227ab80 1 -- 172.21.15.88:0/1699733063 >> 172.21.15.88:0/1699733063 conn(0x55fec8fdd630 msgr2=0x55fec8fe1600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.140+0000 7f537227ab80 1 -- 172.21.15.88:0/1699733063 shutdown_connections 2023-07-14T17:44:29.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.140+0000 7f537227ab80 1 -- 172.21.15.88:0/1699733063 wait complete. 2023-07-14T17:44:29.135 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.140+0000 7f537227ab80 20 client.15459 trim_cache size 0 max 16384 2023-07-14T17:44:29.140 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.157+0000 7f537227ab80 1 Processor -- start 2023-07-14T17:44:29.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.157+0000 7f537227ab80 1 -- start start 2023-07-14T17:44:29.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.157+0000 7f537227ab80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe7f10 0x55fec8fdeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.157+0000 7f537227ab80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe8860 0x55fec90057a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.157+0000 7f537227ab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 0x55fec8fe20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.157+0000 7f537227ab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fec8dc8de0 con 0x55fec8fe9460 2023-07-14T17:44:29.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.157+0000 7f537227ab80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fec8ef7480 con 0x55fec8fe7f10 2023-07-14T17:44:29.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.157+0000 7f537227ab80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fec8ee2470 con 0x55fec8fe8860 2023-07-14T17:44:29.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.158+0000 7f535ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 0x55fec8fe20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.158+0000 7f535ffff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 0x55fec8fe20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43250/0 (socket says 172.21.15.88:43250) 2023-07-14T17:44:29.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.158+0000 7f535ffff700 1 -- 172.21.15.88:0/386619603 learned_addr learned my addr 172.21.15.88:0/386619603 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.158+0000 7f535f7fe700 1 --2- 172.21.15.88:0/386619603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe7f10 0x55fec8fdeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.158+0000 7f535effd700 1 --2- 172.21.15.88:0/386619603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe8860 0x55fec90057a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.158+0000 7f535ffff700 1 -- 172.21.15.88:0/386619603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe8860 msgr2=0x55fec90057a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.158+0000 7f535ffff700 1 --2- 172.21.15.88:0/386619603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe8860 0x55fec90057a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.158+0000 7f535ffff700 1 -- 172.21.15.88:0/386619603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe7f10 msgr2=0x55fec8fdeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.158+0000 7f535ffff700 1 --2- 172.21.15.88:0/386619603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe7f10 0x55fec8fdeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.158+0000 7f535ffff700 1 -- 172.21.15.88:0/386619603 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fec8d8d200 con 0x55fec8fe9460 2023-07-14T17:44:29.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.158+0000 7f535ffff700 1 --2- 172.21.15.88:0/386619603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 0x55fec8fe20a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5358009d10 tx=0x7f53580021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99811af3f01f0f88 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.158+0000 7f53467fc700 1 -- 172.21.15.88:0/386619603 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358025df0 con 0x55fec8fe9460 2023-07-14T17:44:29.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.158+0000 7f53467fc700 1 -- 172.21.15.88:0/386619603 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5358026470 con 0x55fec8fe9460 2023-07-14T17:44:29.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.158+0000 7f53467fc700 1 -- 172.21.15.88:0/386619603 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53580054f0 con 0x55fec8fe9460 2023-07-14T17:44:29.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.159+0000 7f537227ab80 1 -- 172.21.15.88:0/386619603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 msgr2=0x55fec8fe20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.159+0000 7f537227ab80 1 --2- 172.21.15.88:0/386619603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 0x55fec8fe20a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5358009d10 tx=0x7f53580021d0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.159+0000 7f537227ab80 1 -- 172.21.15.88:0/386619603 shutdown_connections 2023-07-14T17:44:29.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.159+0000 7f537227ab80 1 --2- 172.21.15.88:0/386619603 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe9460 0x55fec8fe20a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.159+0000 7f537227ab80 1 --2- 172.21.15.88:0/386619603 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe7f10 0x55fec8fdeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.159+0000 7f537227ab80 1 --2- 172.21.15.88:0/386619603 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe8860 0x55fec90057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.159+0000 7f537227ab80 1 -- 172.21.15.88:0/386619603 >> 172.21.15.88:0/386619603 conn(0x55fec8fdd630 msgr2=0x55fec8ff0650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.159+0000 7f537227ab80 1 -- 172.21.15.88:0/386619603 shutdown_connections 2023-07-14T17:44:29.147 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.159+0000 7f537227ab80 1 -- 172.21.15.88:0/386619603 wait complete. 2023-07-14T17:44:29.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.159+0000 7f537227ab80 1 -- start start 2023-07-14T17:44:29.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.159+0000 7f537227ab80 1 Processor -- start 2023-07-14T17:44:29.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.159+0000 7f537227ab80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe7f10 0x55fec8fee840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.159+0000 7f537227ab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe8860 0x55fec90057a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.159+0000 7f537227ab80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe9460 0x55fec90506b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f537227ab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fec8fef060 con 0x55fec8fe8860 2023-07-14T17:44:29.148 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f537227ab80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fec9050bf0 con 0x55fec8fe7f10 2023-07-14T17:44:29.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f537227ab80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fec9050d30 con 0x55fec8fe9460 2023-07-14T17:44:29.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f537227ab80 10 client.0 _ll_register_callbacks cb 0x7f5358009c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:29.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f535effd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe8860 0x55fec90057a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f535effd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe8860 0x55fec90057a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43254/0 (socket says 172.21.15.88:43254) 2023-07-14T17:44:29.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f535effd700 1 -- 172.21.15.88:0/1056620460 learned_addr learned my addr 172.21.15.88:0/1056620460 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f535ffff700 1 --2- 172.21.15.88:0/1056620460 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe9460 0x55fec90506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f535f7fe700 1 --2- 172.21.15.88:0/1056620460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe7f10 0x55fec8fee840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.149 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f535effd700 1 -- 172.21.15.88:0/1056620460 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe9460 msgr2=0x55fec90506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f535effd700 1 --2- 172.21.15.88:0/1056620460 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe9460 0x55fec90506b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f535effd700 1 -- 172.21.15.88:0/1056620460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe7f10 msgr2=0x55fec8fee840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f535effd700 1 --2- 172.21.15.88:0/1056620460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe7f10 0x55fec8fee840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f535effd700 1 -- 172.21.15.88:0/1056620460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fec8d4fc10 con 0x55fec8fe8860 2023-07-14T17:44:29.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f535ffff700 1 --2- 172.21.15.88:0/1056620460 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe9460 0x55fec90506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:29.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f535effd700 1 --2- 172.21.15.88:0/1056620460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe8860 0x55fec90057a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f5354009990 tx=0x7f5354009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2001566c6f9ce358 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.150 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f533effd700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:29.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f533effd700 1 -- 172.21.15.88:0/1056620460 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5354004910 con 0x55fec8fe8860 2023-07-14T17:44:29.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f533effd700 1 -- 172.21.15.88:0/1056620460 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5354004a70 con 0x55fec8fe8860 2023-07-14T17:44:29.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f537227ab80 1 -- 172.21.15.88:0/1056620460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f535800f050 con 0x55fec8fe8860 2023-07-14T17:44:29.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f533effd700 1 -- 172.21.15.88:0/1056620460 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5354004c30 con 0x55fec8fe8860 2023-07-14T17:44:29.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f537227ab80 10 client.15465 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:29.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.160+0000 7f537227ab80 1 -- 172.21.15.88:0/1056620460 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55fec8d65260 con 0x55fec8fe8860 2023-07-14T17:44:29.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.161+0000 7f533effd700 1 -- 172.21.15.88:0/1056620460 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f535800f050 con 0x55fec8fe8860 2023-07-14T17:44:29.151 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.161+0000 7f533effd700 1 -- 172.21.15.88:0/1056620460 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f535400f460 con 0x55fec8fe8860 2023-07-14T17:44:29.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.161+0000 7f537227ab80 10 client.15465 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:29.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.161+0000 7f537227ab80 10 client.15465 resolve_mds: resolved name 'c' to daemon mds.c 2023-07-14T17:44:29.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.161+0000 7f537227ab80 20 client.15465 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:29.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.161+0000 7f537227ab80 1 --2- 172.21.15.88:0/1056620460 >> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] conn(0x55fec9055b40 0x55fec9052470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.161+0000 7f537227ab80 4 client.15465 mds_command: new command op to 24454 tid=0[{"prefix": "status"}] 2023-07-14T17:44:29.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.161+0000 7f537227ab80 1 -- 172.21.15.88:0/1056620460 --> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fec8f31e60 con 0x55fec9055b40 2023-07-14T17:44:29.152 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.161+0000 7f535f7fe700 1 --2- 172.21.15.88:0/1056620460 >> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] conn(0x55fec9055b40 0x55fec9052470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.161+0000 7f533effd700 10 client.15465 ms_handle_connect on v2:172.21.15.178:6836/3950404084 2023-07-14T17:44:29.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.161+0000 7f535f7fe700 1 --2- 172.21.15.88:0/1056620460 >> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] conn(0x55fec9055b40 0x55fec9052470 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6b26c113ce0386f server_cookie=113dbf0b149c2d84 in_seq=0 out_seq=0 2023-07-14T17:44:29.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f533effd700 1 -- 172.21.15.88:0/1056620460 <== mds.3 v2:172.21.15.178:6836/3950404084 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55fec8f31e60 con 0x55fec9055b40 2023-07-14T17:44:29.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f533effd700 10 client.15465 handle_command_reply: tid=0 2023-07-14T17:44:29.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f537227ab80 1 client.15465 shutdown 2023-07-14T17:44:29.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f537227ab80 1 -- 172.21.15.88:0/1056620460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe8860 msgr2=0x55fec90057a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f537227ab80 1 --2- 172.21.15.88:0/1056620460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe8860 0x55fec90057a0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f5354009990 tx=0x7f5354009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.153 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f537227ab80 1 -- 172.21.15.88:0/1056620460 shutdown_connections 2023-07-14T17:44:29.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f537227ab80 1 --2- 172.21.15.88:0/1056620460 >> [v2:172.21.15.178:6836/3950404084,v1:172.21.15.178:6837/3950404084] conn(0x55fec9055b40 0x55fec9052470 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f537227ab80 1 --2- 172.21.15.88:0/1056620460 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fec8fe9460 0x55fec90506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f533effd700 0 client.15465 ms_handle_reset on v2:172.21.15.178:6836/3950404084 2023-07-14T17:44:29.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f537227ab80 1 --2- 172.21.15.88:0/1056620460 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fec8fe7f10 0x55fec8fee840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f537227ab80 1 --2- 172.21.15.88:0/1056620460 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fec8fe8860 0x55fec90057a0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f537227ab80 1 -- 172.21.15.88:0/1056620460 >> 172.21.15.88:0/1056620460 conn(0x55fec8fdd630 msgr2=0x55fec8fe20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.154 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f537227ab80 1 -- 172.21.15.88:0/1056620460 shutdown_connections 2023-07-14T17:44:29.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f537227ab80 1 -- 172.21.15.88:0/1056620460 wait complete. 2023-07-14T17:44:29.155 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f537227ab80 20 client.15465 trim_cache size 0 max 16384 2023-07-14T17:44:29.155 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-07-14T17:44:29.155 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "b462a556-226c-11ee-9b33-001a4aab830c", 2023-07-14T17:44:29.155 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 3, 2023-07-14T17:44:29.155 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 24454, 2023-07-14T17:44:29.156 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2023-07-14T17:44:29.156 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2023-07-14T17:44:29.156 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2023-07-14T17:44:29.156 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 86.410410666000004, 2023-07-14T17:44:29.156 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 27, 2023-07-14T17:44:29.156 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 88, 2023-07-14T17:44:29.156 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 86, 2023-07-14T17:44:29.157 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 122.792740236 2023-07-14T17:44:29.157 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-07-14T17:44:29.157 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f53660e2700 1 -- 172.21.15.88:0/793306660 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f53480a11d0 msgr2=0x7f53480a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f53660e2700 1 --2- 172.21.15.88:0/793306660 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f53480a11d0 0x7f53480a3680 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f535400c010 tx=0x7f535400b540 comp rx=0 tx=0).stop 2023-07-14T17:44:29.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f53660e2700 1 -- 172.21.15.88:0/793306660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53600feb10 msgr2=0x7f5360197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f53660e2700 1 --2- 172.21.15.88:0/793306660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53600feb10 0x7f5360197ad0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f535000c8f0 tx=0x7f535000ccb0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f53660e2700 1 -- 172.21.15.88:0/793306660 shutdown_connections 2023-07-14T17:44:29.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f53660e2700 1 --2- 172.21.15.88:0/793306660 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f5360100690 0x7f536019f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f53660e2700 1 --2- 172.21.15.88:0/793306660 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f53480a11d0 0x7f53480a3680 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.158 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f53660e2700 1 --2- 172.21.15.88:0/793306660 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53600feb10 0x7f5360197ad0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f53660e2700 1 --2- 172.21.15.88:0/793306660 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f53600ffd00 0x7f5360198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.162+0000 7f53660e2700 1 -- 172.21.15.88:0/793306660 >> 172.21.15.88:0/793306660 conn(0x7f53600fa130 msgr2=0x7f53600677e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.163+0000 7f53660e2700 1 -- 172.21.15.88:0/793306660 shutdown_connections 2023-07-14T17:44:29.159 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.163+0000 7f53660e2700 1 -- 172.21.15.88:0/793306660 wait complete. 2023-07-14T17:44:29.159 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:29 smithi088 ceph-mon[127465]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2023-07-14T17:44:29.159 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:29 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2917064616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:44:29.160 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:29 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2110490728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:29.160 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:29 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1271041221' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:29.161 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-07-14T17:44:29.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.299+0000 7f89ea661700 1 -- 172.21.15.88:0/2129951701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89e40febc0 msgr2=0x7f89e40fefd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.299+0000 7f89ea661700 1 --2- 172.21.15.88:0/2129951701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89e40febc0 0x7f89e40fefd0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f89d4009990 tx=0x7f89d4009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.299+0000 7f89ea661700 1 -- 172.21.15.88:0/2129951701 shutdown_connections 2023-07-14T17:44:29.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.299+0000 7f89ea661700 1 --2- 172.21.15.88:0/2129951701 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89e4100740 0x7f89e4107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.299+0000 7f89ea661700 1 --2- 172.21.15.88:0/2129951701 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89e40febc0 0x7f89e40fefd0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.299+0000 7f89ea661700 1 --2- 172.21.15.88:0/2129951701 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89e40ffdb0 0x7f89e4100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.299+0000 7f89ea661700 1 -- 172.21.15.88:0/2129951701 >> 172.21.15.88:0/2129951701 conn(0x7f89e40fa200 msgr2=0x7f89e40fc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.300+0000 7f89ea661700 1 -- 172.21.15.88:0/2129951701 shutdown_connections 2023-07-14T17:44:29.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.300+0000 7f89ea661700 1 -- 172.21.15.88:0/2129951701 wait complete. 2023-07-14T17:44:29.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.300+0000 7f89ea661700 1 Processor -- start 2023-07-14T17:44:29.284 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.300+0000 7f89ea661700 1 -- start start 2023-07-14T17:44:29.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.300+0000 7f89ea661700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89e40febc0 0x7f89e4069520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.300+0000 7f89ea661700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89e40ffdb0 0x7f89e419a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.300+0000 7f89ea661700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89e4100740 0x7f89e419cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.300+0000 7f89ea661700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e419d3f0 con 0x7f89e40ffdb0 2023-07-14T17:44:29.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.300+0000 7f89ea661700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e419d560 con 0x7f89e40febc0 2023-07-14T17:44:29.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.300+0000 7f89ea661700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89e419d720 con 0x7f89e4100740 2023-07-14T17:44:29.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.301+0000 7f89e37fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89e40ffdb0 0x7f89e419a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.301+0000 7f89e37fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89e40ffdb0 0x7f89e419a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43268/0 (socket says 172.21.15.88:43268) 2023-07-14T17:44:29.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.301+0000 7f89e37fe700 1 -- 172.21.15.88:0/1632800845 learned_addr learned my addr 172.21.15.88:0/1632800845 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.301+0000 7f89e3fff700 1 --2- 172.21.15.88:0/1632800845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89e40febc0 0x7f89e4069520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.301+0000 7f89e8bfe700 1 --2- 172.21.15.88:0/1632800845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89e4100740 0x7f89e419cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.301+0000 7f89e37fe700 1 -- 172.21.15.88:0/1632800845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89e4100740 msgr2=0x7f89e419cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.301+0000 7f89e37fe700 1 --2- 172.21.15.88:0/1632800845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89e4100740 0x7f89e419cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.301+0000 7f89e37fe700 1 -- 172.21.15.88:0/1632800845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89e40febc0 msgr2=0x7f89e4069520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.301+0000 7f89e37fe700 1 --2- 172.21.15.88:0/1632800845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89e40febc0 0x7f89e4069520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.301+0000 7f89e37fe700 1 -- 172.21.15.88:0/1632800845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d40095f0 con 0x7f89e40ffdb0 2023-07-14T17:44:29.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.301+0000 7f89e8bfe700 1 --2- 172.21.15.88:0/1632800845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89e4100740 0x7f89e419cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:29.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.301+0000 7f89e37fe700 1 --2- 172.21.15.88:0/1632800845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89e40ffdb0 0x7f89e419a610 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f89d800d950 tx=0x7f89d800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.301+0000 7f89e17fa700 1 -- 172.21.15.88:0/1632800845 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d8009b40 con 0x7f89e40ffdb0 2023-07-14T17:44:29.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.301+0000 7f89e17fa700 1 -- 172.21.15.88:0/1632800845 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f89d8009ca0 con 0x7f89e40ffdb0 2023-07-14T17:44:29.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.301+0000 7f89ea661700 1 -- 172.21.15.88:0/1632800845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89e419da00 con 0x7f89e40ffdb0 2023-07-14T17:44:29.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.301+0000 7f89e17fa700 1 -- 172.21.15.88:0/1632800845 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d800f620 con 0x7f89e40ffdb0 2023-07-14T17:44:29.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.301+0000 7f89ea661700 1 -- 172.21.15.88:0/1632800845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e419df10 con 0x7f89e40ffdb0 2023-07-14T17:44:29.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.303+0000 7f89e17fa700 1 -- 172.21.15.88:0/1632800845 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f89d8010460 con 0x7f89e40ffdb0 2023-07-14T17:44:29.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.304+0000 7f89e17fa700 1 --2- 172.21.15.88:0/1632800845 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f89cc0a1150 0x7f89cc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.304+0000 7f89e3fff700 1 --2- 172.21.15.88:0/1632800845 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f89cc0a1150 0x7f89cc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.304+0000 7f89e17fa700 1 -- 172.21.15.88:0/1632800845 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f89d80ccf80 con 0x7f89e40ffdb0 2023-07-14T17:44:29.294 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.304+0000 7f89e3fff700 1 --2- 172.21.15.88:0/1632800845 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f89cc0a1150 0x7f89cc0a3600 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f89d400b5c0 tx=0x7f89d4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89f67f9b80 1 Processor -- start 2023-07-14T17:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89f67f9b80 1 -- start start 2023-07-14T17:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89f67f9b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803db9f10 0x556803dba320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89f67f9b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dbae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89f67f9b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803dbb460 0x556803dbb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89f67f9b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556803cbec60 con 0x556803dbb460 2023-07-14T17:44:29.295 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89f67f9b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x556803d119f0 con 0x556803dba860 2023-07-14T17:44:29.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89f67f9b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x556803d0ae80 con 0x556803db9f10 2023-07-14T17:44:29.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89e8bfe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803dbb460 0x556803dbb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89e8bfe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803dbb460 0x556803dbb8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43278/0 (socket says 172.21.15.88:43278) 2023-07-14T17:44:29.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89e8bfe700 1 -- 172.21.15.88:0/4247244954 learned_addr learned my addr 172.21.15.88:0/4247244954 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89e3fff700 1 --2- 172.21.15.88:0/4247244954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803db9f10 0x556803dba320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89e37fe700 1 --2- 172.21.15.88:0/4247244954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dbae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.296 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89e8bfe700 1 -- 172.21.15.88:0/4247244954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803db9f10 msgr2=0x556803dba320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89e8bfe700 1 --2- 172.21.15.88:0/4247244954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803db9f10 0x556803dba320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89e8bfe700 1 -- 172.21.15.88:0/4247244954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 msgr2=0x556803dbae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89e8bfe700 1 --2- 172.21.15.88:0/4247244954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dbae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89e8bfe700 1 -- 172.21.15.88:0/4247244954 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556803b38d90 con 0x556803dbb460 2023-07-14T17:44:29.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89e37fe700 1 --2- 172.21.15.88:0/4247244954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dbae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:29.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.305+0000 7f89e8bfe700 1 --2- 172.21.15.88:0/4247244954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803dbb460 0x556803dbb8d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009d50 tx=0x7f89dc0077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc6fa7255ad1f3f1 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.306+0000 7f89ca7fc700 1 -- 172.21.15.88:0/4247244954 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc014600 con 0x556803dbb460 2023-07-14T17:44:29.297 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.306+0000 7f89ca7fc700 1 -- 172.21.15.88:0/4247244954 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f89dc014bf0 con 0x556803dbb460 2023-07-14T17:44:29.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.306+0000 7f89ca7fc700 1 -- 172.21.15.88:0/4247244954 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc013a90 con 0x556803dbb460 2023-07-14T17:44:29.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.306+0000 7f89f67f9b80 1 -- 172.21.15.88:0/4247244954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803dbb460 msgr2=0x556803dbb8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.306+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/4247244954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803dbb460 0x556803dbb8d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f89dc009d50 tx=0x7f89dc0077e0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.306+0000 7f89f67f9b80 1 -- 172.21.15.88:0/4247244954 shutdown_connections 2023-07-14T17:44:29.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.306+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/4247244954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803dbb460 0x556803dbb8d0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.306+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/4247244954 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803db9f10 0x556803dba320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.306+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/4247244954 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dbae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.298 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.306+0000 7f89f67f9b80 1 -- 172.21.15.88:0/4247244954 >> 172.21.15.88:0/4247244954 conn(0x556803daf630 msgr2=0x556803db4980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.306+0000 7f89f67f9b80 1 -- 172.21.15.88:0/4247244954 shutdown_connections 2023-07-14T17:44:29.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.306+0000 7f89f67f9b80 1 -- 172.21.15.88:0/4247244954 wait complete. 2023-07-14T17:44:29.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.307+0000 7f89f67f9b80 1 -- start start 2023-07-14T17:44:29.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.307+0000 7f89f67f9b80 1 Processor -- start 2023-07-14T17:44:29.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.307+0000 7f89f67f9b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803db9f10 0x556803dc3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.307+0000 7f89f67f9b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dc3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.299 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.307+0000 7f89f67f9b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803dbb460 0x556803dcaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.307+0000 7f89f67f9b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556803caaa70 con 0x556803dbb460 2023-07-14T17:44:29.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.307+0000 7f89f67f9b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x556803caa770 con 0x556803dba860 2023-07-14T17:44:29.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.307+0000 7f89f67f9b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x556803b8e8d0 con 0x556803db9f10 2023-07-14T17:44:29.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.307+0000 7f89f67f9b80 10 client.0 _ll_register_callbacks cb 0x556803ca8710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:29.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.307+0000 7f89e8bfe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803dbb460 0x556803dcaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.307+0000 7f89e8bfe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803dbb460 0x556803dcaef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43284/0 (socket says 172.21.15.88:43284) 2023-07-14T17:44:29.300 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.307+0000 7f89e8bfe700 1 -- 172.21.15.88:0/1281490540 learned_addr learned my addr 172.21.15.88:0/1281490540 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.307+0000 7f89e37fe700 1 --2- 172.21.15.88:0/1281490540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dc3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.307+0000 7f89e3fff700 1 --2- 172.21.15.88:0/1281490540 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803db9f10 0x556803dc3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.308+0000 7f89e8bfe700 1 -- 172.21.15.88:0/1281490540 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803db9f10 msgr2=0x556803dc3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.308+0000 7f89e8bfe700 1 --2- 172.21.15.88:0/1281490540 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803db9f10 0x556803dc3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.308+0000 7f89e8bfe700 1 -- 172.21.15.88:0/1281490540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 msgr2=0x556803dc3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.308+0000 7f89e8bfe700 1 --2- 172.21.15.88:0/1281490540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dc3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.301 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.308+0000 7f89e8bfe700 1 -- 172.21.15.88:0/1281490540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89dc009980 con 0x556803dbb460 2023-07-14T17:44:29.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.308+0000 7f89e8bfe700 1 --2- 172.21.15.88:0/1281490540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803dbb460 0x556803dcaef0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f89dc0060b0 tx=0x7f89dc0060e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c55fd5e71835a01 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.308+0000 7f89c37fe700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:29.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.308+0000 7f89c37fe700 1 -- 172.21.15.88:0/1281490540 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89dc014910 con 0x556803dbb460 2023-07-14T17:44:29.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.308+0000 7f89c37fe700 1 -- 172.21.15.88:0/1281490540 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f89dc0137b0 con 0x556803dbb460 2023-07-14T17:44:29.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.308+0000 7f89f67f9b80 1 -- 172.21.15.88:0/1281490540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556803c644a0 con 0x556803dbb460 2023-07-14T17:44:29.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.308+0000 7f89c37fe700 1 -- 172.21.15.88:0/1281490540 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89dc014a70 con 0x556803dbb460 2023-07-14T17:44:29.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.308+0000 7f89f67f9b80 10 client.15477 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:29.302 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.308+0000 7f89f67f9b80 1 -- 172.21.15.88:0/1281490540 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x556803b21c10 con 0x556803dbb460 2023-07-14T17:44:29.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.308+0000 7f89c37fe700 1 -- 172.21.15.88:0/1281490540 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x556803c644a0 con 0x556803dbb460 2023-07-14T17:44:29.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.308+0000 7f89c37fe700 1 -- 172.21.15.88:0/1281490540 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f89dc005530 con 0x556803dbb460 2023-07-14T17:44:29.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.309+0000 7f89f67f9b80 10 client.15477 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:29.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.309+0000 7f89f67f9b80 10 client.15477 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T17:44:29.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.309+0000 7f89f67f9b80 20 client.15477 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:29.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.309+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/1281490540 >> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] conn(0x556803dceb30 0x556803dcdf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.303 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.309+0000 7f89f67f9b80 4 client.15477 mds_command: new command op to 24469 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T17:44:29.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.309+0000 7f89f67f9b80 1 -- 172.21.15.88:0/1281490540 --> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556803d88d50 con 0x556803dceb30 2023-07-14T17:44:29.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.309+0000 7f89e3fff700 1 --2- 172.21.15.88:0/1281490540 >> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] conn(0x556803dceb30 0x556803dcdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.309+0000 7f89c37fe700 10 client.15477 ms_handle_connect on v2:172.21.15.178:6838/2953415077 2023-07-14T17:44:29.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.309+0000 7f89e3fff700 1 --2- 172.21.15.88:0/1281490540 >> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] conn(0x556803dceb30 0x556803dcdf00 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=430835cd13e8bf9d server_cookie=2638d0381ef982b1 in_seq=0 out_seq=0 2023-07-14T17:44:29.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.310+0000 7f89c37fe700 1 -- 172.21.15.88:0/1281490540 <== mds.2 v2:172.21.15.178:6838/2953415077 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x556803d88d50 con 0x556803dceb30 2023-07-14T17:44:29.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.310+0000 7f89c37fe700 10 client.15477 handle_command_reply: tid=0 2023-07-14T17:44:29.304 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.310+0000 7f89f67f9b80 1 client.15477 shutdown 2023-07-14T17:44:29.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.310+0000 7f89f67f9b80 1 -- 172.21.15.88:0/1281490540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803dbb460 msgr2=0x556803dcaef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.310+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/1281490540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803dbb460 0x556803dcaef0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f89dc0060b0 tx=0x7f89dc0060e0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.310+0000 7f89f67f9b80 1 -- 172.21.15.88:0/1281490540 shutdown_connections 2023-07-14T17:44:29.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.310+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/1281490540 >> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] conn(0x556803dceb30 0x556803dcdf00 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.310+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/1281490540 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803dbb460 0x556803dcaef0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.310+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/1281490540 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803db9f10 0x556803dc3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.310+0000 7f89c37fe700 0 client.15477 ms_handle_reset on v2:172.21.15.178:6838/2953415077 2023-07-14T17:44:29.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.310+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/1281490540 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dc3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.310+0000 7f89f67f9b80 1 -- 172.21.15.88:0/1281490540 >> 172.21.15.88:0/1281490540 conn(0x556803daf630 msgr2=0x556803db3600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.310+0000 7f89f67f9b80 1 -- 172.21.15.88:0/1281490540 shutdown_connections 2023-07-14T17:44:29.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.310+0000 7f89f67f9b80 1 -- 172.21.15.88:0/1281490540 wait complete. 2023-07-14T17:44:29.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.310+0000 7f89f67f9b80 20 client.15477 trim_cache size 0 max 16384 2023-07-14T17:44:29.311 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.327+0000 7f89f67f9b80 1 Processor -- start 2023-07-14T17:44:29.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89f67f9b80 1 -- start start 2023-07-14T17:44:29.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89f67f9b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803db9f10 0x556803dc2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89f67f9b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dcaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89f67f9b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803dbb460 0x556803dcdf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89f67f9b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556803b9ade0 con 0x556803db9f10 2023-07-14T17:44:29.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89f67f9b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x556803cc9480 con 0x556803dba860 2023-07-14T17:44:29.312 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89f67f9b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x556803ca6d50 con 0x556803dbb460 2023-07-14T17:44:29.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89e3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803db9f10 0x556803dc2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89e3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803db9f10 0x556803dc2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43288/0 (socket says 172.21.15.88:43288) 2023-07-14T17:44:29.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89e3fff700 1 -- 172.21.15.88:0/3984390634 learned_addr learned my addr 172.21.15.88:0/3984390634 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89e8bfe700 1 --2- 172.21.15.88:0/3984390634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803dbb460 0x556803dcdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89e37fe700 1 --2- 172.21.15.88:0/3984390634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dcaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89e3fff700 1 -- 172.21.15.88:0/3984390634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803dbb460 msgr2=0x556803dcdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.313 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89e3fff700 1 --2- 172.21.15.88:0/3984390634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803dbb460 0x556803dcdf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89e3fff700 1 -- 172.21.15.88:0/3984390634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 msgr2=0x556803dcaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89e3fff700 1 --2- 172.21.15.88:0/3984390634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dcaeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89e3fff700 1 -- 172.21.15.88:0/3984390634 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556803b5f200 con 0x556803db9f10 2023-07-14T17:44:29.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89e37fe700 1 --2- 172.21.15.88:0/3984390634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dcaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:29.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89e3fff700 1 --2- 172.21.15.88:0/3984390634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803db9f10 0x556803dc2680 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f89d4005c50 tx=0x7f89d4022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c3b26fc6e23ed8 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89caffd700 1 -- 172.21.15.88:0/3984390634 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d4034250 con 0x556803db9f10 2023-07-14T17:44:29.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89caffd700 1 -- 172.21.15.88:0/3984390634 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f89d402c040 con 0x556803db9f10 2023-07-14T17:44:29.314 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.328+0000 7f89caffd700 1 -- 172.21.15.88:0/3984390634 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d4005500 con 0x556803db9f10 2023-07-14T17:44:29.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.329+0000 7f89f67f9b80 1 -- 172.21.15.88:0/3984390634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803db9f10 msgr2=0x556803dc2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.329+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/3984390634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803db9f10 0x556803dc2680 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f89d4005c50 tx=0x7f89d4022000 comp rx=0 tx=0).stop 2023-07-14T17:44:29.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.329+0000 7f89f67f9b80 1 -- 172.21.15.88:0/3984390634 shutdown_connections 2023-07-14T17:44:29.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.329+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/3984390634 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803dbb460 0x556803dcdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.329+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/3984390634 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803db9f10 0x556803dc2680 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.329+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/3984390634 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dcaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.329+0000 7f89f67f9b80 1 -- 172.21.15.88:0/3984390634 >> 172.21.15.88:0/3984390634 conn(0x556803daf630 msgr2=0x556803db40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.315 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.329+0000 7f89f67f9b80 1 -- 172.21.15.88:0/3984390634 shutdown_connections 2023-07-14T17:44:29.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.329+0000 7f89f67f9b80 1 -- 172.21.15.88:0/3984390634 wait complete. 2023-07-14T17:44:29.316 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.329+0000 7f89f67f9b80 1 -- start start 2023-07-14T17:44:29.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89f67f9b80 1 Processor -- start 2023-07-14T17:44:29.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89f67f9b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803db9f10 0x556803dc0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89f67f9b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dcaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89f67f9b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803dbb460 0x556803e22800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89f67f9b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556803e22f40 con 0x556803db9f10 2023-07-14T17:44:29.319 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89f67f9b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x556803e230b0 con 0x556803dba860 2023-07-14T17:44:29.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89f67f9b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x556803e23270 con 0x556803dbb460 2023-07-14T17:44:29.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89f67f9b80 10 client.0 _ll_register_callbacks cb 0x7f89dc009ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:29.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89e3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803db9f10 0x556803dc0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89e3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803db9f10 0x556803dc0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43294/0 (socket says 172.21.15.88:43294) 2023-07-14T17:44:29.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89e3fff700 1 -- 172.21.15.88:0/1047170474 learned_addr learned my addr 172.21.15.88:0/1047170474 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89e8bfe700 1 --2- 172.21.15.88:0/1047170474 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803dbb460 0x556803e22800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.320 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89e37fe700 1 --2- 172.21.15.88:0/1047170474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dcaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89e3fff700 1 -- 172.21.15.88:0/1047170474 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803dbb460 msgr2=0x556803e22800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89e3fff700 1 --2- 172.21.15.88:0/1047170474 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803dbb460 0x556803e22800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89e3fff700 1 -- 172.21.15.88:0/1047170474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 msgr2=0x556803dcaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89e3fff700 1 --2- 172.21.15.88:0/1047170474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dcaeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89e3fff700 1 -- 172.21.15.88:0/1047170474 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89e4107930 con 0x556803db9f10 2023-07-14T17:44:29.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89e8bfe700 1 --2- 172.21.15.88:0/1047170474 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803dbb460 0x556803e22800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:29.321 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89e37fe700 1 --2- 172.21.15.88:0/1047170474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dcaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:29.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89e3fff700 1 --2- 172.21.15.88:0/1047170474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803db9f10 0x556803dc0d00 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f89d4005080 tx=0x7f89d4009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f6bad04eeaa56cc server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89c37fe700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:29.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89c37fe700 1 -- 172.21.15.88:0/1047170474 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d4034460 con 0x556803db9f10 2023-07-14T17:44:29.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89c37fe700 1 -- 172.21.15.88:0/1047170474 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f89d40056a0 con 0x556803db9f10 2023-07-14T17:44:29.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.330+0000 7f89f67f9b80 1 -- 172.21.15.88:0/1047170474 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f89dc014bf0 con 0x556803db9f10 2023-07-14T17:44:29.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.331+0000 7f89c37fe700 1 -- 172.21.15.88:0/1047170474 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89d4034940 con 0x556803db9f10 2023-07-14T17:44:29.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.331+0000 7f89f67f9b80 10 client.15483 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:29.322 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.331+0000 7f89f67f9b80 1 -- 172.21.15.88:0/1047170474 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x556803b37260 con 0x556803db9f10 2023-07-14T17:44:29.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.331+0000 7f89c37fe700 1 -- 172.21.15.88:0/1047170474 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f89dc014bf0 con 0x556803db9f10 2023-07-14T17:44:29.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.331+0000 7f89c37fe700 1 -- 172.21.15.88:0/1047170474 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f89d4004cb0 con 0x556803db9f10 2023-07-14T17:44:29.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.331+0000 7f89f67f9b80 10 client.15483 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:29.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.331+0000 7f89f67f9b80 10 client.15483 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T17:44:29.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.331+0000 7f89f67f9b80 20 client.15483 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:29.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.331+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/1047170474 >> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] conn(0x556803e27ac0 0x556803e24410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.323 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.331+0000 7f89f67f9b80 4 client.15483 mds_command: new command op to 24469 tid=0[{"prefix": "status"}] 2023-07-14T17:44:29.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.331+0000 7f89f67f9b80 1 -- 172.21.15.88:0/1047170474 --> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556803d03e60 con 0x556803e27ac0 2023-07-14T17:44:29.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.331+0000 7f89e37fe700 1 --2- 172.21.15.88:0/1047170474 >> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] conn(0x556803e27ac0 0x556803e24410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.332+0000 7f89c37fe700 10 client.15483 ms_handle_connect on v2:172.21.15.178:6838/2953415077 2023-07-14T17:44:29.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.332+0000 7f89e37fe700 1 --2- 172.21.15.88:0/1047170474 >> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] conn(0x556803e27ac0 0x556803e24410 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=603d09e48db67483 server_cookie=98f8831762ed53bf in_seq=0 out_seq=0 2023-07-14T17:44:29.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.332+0000 7f89c37fe700 1 -- 172.21.15.88:0/1047170474 <== mds.2 v2:172.21.15.178:6838/2953415077 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x556803d03e60 con 0x556803e27ac0 2023-07-14T17:44:29.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.332+0000 7f89c37fe700 10 client.15483 handle_command_reply: tid=0 2023-07-14T17:44:29.324 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.332+0000 7f89f67f9b80 1 client.15483 shutdown 2023-07-14T17:44:29.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.332+0000 7f89f67f9b80 1 -- 172.21.15.88:0/1047170474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803db9f10 msgr2=0x556803dc0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.332+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/1047170474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803db9f10 0x556803dc0d00 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f89d4005080 tx=0x7f89d4009310 comp rx=0 tx=0).stop 2023-07-14T17:44:29.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.332+0000 7f89f67f9b80 1 -- 172.21.15.88:0/1047170474 shutdown_connections 2023-07-14T17:44:29.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.332+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/1047170474 >> [v2:172.21.15.178:6838/2953415077,v1:172.21.15.178:6839/2953415077] conn(0x556803e27ac0 0x556803e24410 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.332+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/1047170474 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x556803dbb460 0x556803e22800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.332+0000 7f89c37fe700 0 client.15483 ms_handle_reset on v2:172.21.15.178:6838/2953415077 2023-07-14T17:44:29.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.332+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/1047170474 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556803db9f10 0x556803dc0d00 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.325 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.332+0000 7f89f67f9b80 1 --2- 172.21.15.88:0/1047170474 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x556803dba860 0x556803dcaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.332+0000 7f89f67f9b80 1 -- 172.21.15.88:0/1047170474 >> 172.21.15.88:0/1047170474 conn(0x556803daf630 msgr2=0x556803dcdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.332+0000 7f89f67f9b80 1 -- 172.21.15.88:0/1047170474 shutdown_connections 2023-07-14T17:44:29.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.332+0000 7f89f67f9b80 1 -- 172.21.15.88:0/1047170474 wait complete. 2023-07-14T17:44:29.326 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.332+0000 7f89f67f9b80 20 client.15483 trim_cache size 0 max 16384 2023-07-14T17:44:29.326 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-07-14T17:44:29.327 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "b462a556-226c-11ee-9b33-001a4aab830c", 2023-07-14T17:44:29.327 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 2, 2023-07-14T17:44:29.327 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 24469, 2023-07-14T17:44:29.327 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2023-07-14T17:44:29.327 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2023-07-14T17:44:29.327 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2023-07-14T17:44:29.327 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 87.588515997000002, 2023-07-14T17:44:29.328 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 27, 2023-07-14T17:44:29.328 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 88, 2023-07-14T17:44:29.328 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 86, 2023-07-14T17:44:29.328 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 118.301471667 2023-07-14T17:44:29.328 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-07-14T17:44:29.329 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.333+0000 7f89ea661700 1 -- 172.21.15.88:0/1632800845 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f89cc0a1150 msgr2=0x7f89cc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.333+0000 7f89ea661700 1 --2- 172.21.15.88:0/1632800845 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f89cc0a1150 0x7f89cc0a3600 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f89d400b5c0 tx=0x7f89d4005fb0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.333+0000 7f89ea661700 1 -- 172.21.15.88:0/1632800845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89e40ffdb0 msgr2=0x7f89e419a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.333+0000 7f89ea661700 1 --2- 172.21.15.88:0/1632800845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89e40ffdb0 0x7f89e419a610 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f89d800d950 tx=0x7f89d800dd10 comp rx=0 tx=0).stop 2023-07-14T17:44:29.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.333+0000 7f89ea661700 1 -- 172.21.15.88:0/1632800845 shutdown_connections 2023-07-14T17:44:29.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.333+0000 7f89ea661700 1 --2- 172.21.15.88:0/1632800845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f89e4100740 0x7f89e419cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.333+0000 7f89ea661700 1 --2- 172.21.15.88:0/1632800845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f89e40febc0 0x7f89e4069520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.330 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.333+0000 7f89ea661700 1 --2- 172.21.15.88:0/1632800845 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f89cc0a1150 0x7f89cc0a3600 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.333+0000 7f89ea661700 1 --2- 172.21.15.88:0/1632800845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89e40ffdb0 0x7f89e419a610 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.333+0000 7f89ea661700 1 -- 172.21.15.88:0/1632800845 >> 172.21.15.88:0/1632800845 conn(0x7f89e40fa200 msgr2=0x7f89e40fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.333+0000 7f89ea661700 1 -- 172.21.15.88:0/1632800845 shutdown_connections 2023-07-14T17:44:29.331 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.333+0000 7f89ea661700 1 -- 172.21.15.88:0/1632800845 wait complete. 2023-07-14T17:44:29.331 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-07-14T17:44:29.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:29 smithi178 ceph-mon[129368]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2023-07-14T17:44:29.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:29 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2917064616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:44:29.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:29 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2110490728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:29.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:29 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1271041221' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:29.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.471+0000 7f0d02d0f700 1 -- 172.21.15.88:0/1530440731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cfc0feb10 msgr2=0x7f0cfc0fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.455 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.471+0000 7f0d02d0f700 1 --2- 172.21.15.88:0/1530440731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cfc0feb10 0x7f0cfc0fef20 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009990 tx=0x7f0ce4009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.471+0000 7f0d02d0f700 1 -- 172.21.15.88:0/1530440731 shutdown_connections 2023-07-14T17:44:29.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.471+0000 7f0d02d0f700 1 --2- 172.21.15.88:0/1530440731 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cfc100690 0x7f0cfc106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.471+0000 7f0d02d0f700 1 --2- 172.21.15.88:0/1530440731 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cfc0feb10 0x7f0cfc0fef20 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.471+0000 7f0d02d0f700 1 --2- 172.21.15.88:0/1530440731 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cfc0ffd00 0x7f0cfc100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.471+0000 7f0d02d0f700 1 -- 172.21.15.88:0/1530440731 >> 172.21.15.88:0/1530440731 conn(0x7f0cfc0fa130 msgr2=0x7f0cfc0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.456 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.471+0000 7f0d02d0f700 1 -- 172.21.15.88:0/1530440731 shutdown_connections 2023-07-14T17:44:29.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.471+0000 7f0d02d0f700 1 -- 172.21.15.88:0/1530440731 wait complete. 2023-07-14T17:44:29.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.472+0000 7f0d02d0f700 1 Processor -- start 2023-07-14T17:44:29.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.472+0000 7f0d02d0f700 1 -- start start 2023-07-14T17:44:29.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.472+0000 7f0d02d0f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cfc0feb10 0x7f0cfc194830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.472+0000 7f0d02d0f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cfc0ffd00 0x7f0cfc194d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.472+0000 7f0d02d0f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cfc100690 0x7f0cfc19bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.457 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.472+0000 7f0d02d0f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc19c3b0 con 0x7f0cfc100690 2023-07-14T17:44:29.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.472+0000 7f0d02d0f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc19c4f0 con 0x7f0cfc0feb10 2023-07-14T17:44:29.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.472+0000 7f0d02d0f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc19c680 con 0x7f0cfc0ffd00 2023-07-14T17:44:29.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.472+0000 7f0d012ac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cfc100690 0x7f0cfc19bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.472+0000 7f0d012ac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cfc100690 0x7f0cfc19bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43310/0 (socket says 172.21.15.88:43310) 2023-07-14T17:44:29.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.472+0000 7f0d00aab700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cfc0feb10 0x7f0cfc194830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.472+0000 7f0d012ac700 1 -- 172.21.15.88:0/148568654 learned_addr learned my addr 172.21.15.88:0/148568654 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.458 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.472+0000 7f0cfbfff700 1 --2- 172.21.15.88:0/148568654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cfc0ffd00 0x7f0cfc194d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.472+0000 7f0d012ac700 1 -- 172.21.15.88:0/148568654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cfc0ffd00 msgr2=0x7f0cfc194d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.472+0000 7f0d012ac700 1 --2- 172.21.15.88:0/148568654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cfc0ffd00 0x7f0cfc194d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.472+0000 7f0d012ac700 1 -- 172.21.15.88:0/148568654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cfc0feb10 msgr2=0x7f0cfc194830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.472+0000 7f0d012ac700 1 --2- 172.21.15.88:0/148568654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cfc0feb10 0x7f0cfc194830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.472+0000 7f0d012ac700 1 -- 172.21.15.88:0/148568654 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce40095f0 con 0x7f0cfc100690 2023-07-14T17:44:29.459 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.473+0000 7f0d012ac700 1 --2- 172.21.15.88:0/148568654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cfc100690 0x7f0cfc19bde0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4007ee0 tx=0x7f0cf400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.473+0000 7f0cf9ffb700 1 -- 172.21.15.88:0/148568654 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4017070 con 0x7f0cfc100690 2023-07-14T17:44:29.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.473+0000 7f0cf9ffb700 1 -- 172.21.15.88:0/148568654 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0cf400f040 con 0x7f0cfc100690 2023-07-14T17:44:29.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.473+0000 7f0d02d0f700 1 -- 172.21.15.88:0/148568654 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cfc19c960 con 0x7f0cfc100690 2023-07-14T17:44:29.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.473+0000 7f0cf9ffb700 1 -- 172.21.15.88:0/148568654 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf4005570 con 0x7f0cfc100690 2023-07-14T17:44:29.460 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.473+0000 7f0d02d0f700 1 -- 172.21.15.88:0/148568654 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cfc19ce70 con 0x7f0cfc100690 2023-07-14T17:44:29.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.475+0000 7f0cf9ffb700 1 -- 172.21.15.88:0/148568654 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0cf400db20 con 0x7f0cfc100690 2023-07-14T17:44:29.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.475+0000 7f0cf9ffb700 1 --2- 172.21.15.88:0/148568654 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0ce80a1150 0x7f0ce80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.475+0000 7f0cf9ffb700 1 -- 172.21.15.88:0/148568654 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0cf40cc2b0 con 0x7f0cfc100690 2023-07-14T17:44:29.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.475+0000 7f0d00aab700 1 --2- 172.21.15.88:0/148568654 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0ce80a1150 0x7f0ce80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.476+0000 7f0d00aab700 1 --2- 172.21.15.88:0/148568654 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0ce80a1150 0x7f0ce80a3600 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40098e0 tx=0x7f0ce400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.476+0000 7f0d0eea7b80 1 Processor -- start 2023-07-14T17:44:29.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.476+0000 7f0d0eea7b80 1 -- start start 2023-07-14T17:44:29.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0d0eea7b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb35f10 0x55fe0cb36320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0d0eea7b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb36860 0x55fe0cb36e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.467 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0d0eea7b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb37460 0x55fe0cb378d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0d0eea7b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe0ca3ac60 con 0x55fe0cb36860 2023-07-14T17:44:29.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0d0eea7b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe0ca8d9f0 con 0x55fe0cb35f10 2023-07-14T17:44:29.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0d0eea7b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe0ca86e80 con 0x55fe0cb37460 2023-07-14T17:44:29.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0cfbfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb36860 0x55fe0cb36e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0cfbfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb36860 0x55fe0cb36e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43316/0 (socket says 172.21.15.88:43316) 2023-07-14T17:44:29.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0cfbfff700 1 -- 172.21.15.88:0/916326454 learned_addr learned my addr 172.21.15.88:0/916326454 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0d00aab700 1 --2- 172.21.15.88:0/916326454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb35f10 0x55fe0cb36320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0d012ac700 1 --2- 172.21.15.88:0/916326454 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb37460 0x55fe0cb378d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0cfbfff700 1 -- 172.21.15.88:0/916326454 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb37460 msgr2=0x55fe0cb378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0cfbfff700 1 --2- 172.21.15.88:0/916326454 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb37460 0x55fe0cb378d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0cfbfff700 1 -- 172.21.15.88:0/916326454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb35f10 msgr2=0x55fe0cb36320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0cfbfff700 1 --2- 172.21.15.88:0/916326454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb35f10 0x55fe0cb36320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0cfbfff700 1 -- 172.21.15.88:0/916326454 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe0c8b4d90 con 0x55fe0cb36860 2023-07-14T17:44:29.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0cfbfff700 1 --2- 172.21.15.88:0/916326454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb36860 0x55fe0cb36e80 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0cec009d80 tx=0x7f0cec011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7240ee6bc0c7eca9 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0cf2ffd700 1 -- 172.21.15.88:0/916326454 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec004230 con 0x55fe0cb36860 2023-07-14T17:44:29.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0cf2ffd700 1 -- 172.21.15.88:0/916326454 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0cec004390 con 0x55fe0cb36860 2023-07-14T17:44:29.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.477+0000 7f0cf2ffd700 1 -- 172.21.15.88:0/916326454 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec0197e0 con 0x55fe0cb36860 2023-07-14T17:44:29.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.478+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/916326454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb36860 msgr2=0x55fe0cb36e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.478+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/916326454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb36860 0x55fe0cb36e80 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0cec009d80 tx=0x7f0cec011720 comp rx=0 tx=0).stop 2023-07-14T17:44:29.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.478+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/916326454 shutdown_connections 2023-07-14T17:44:29.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.478+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/916326454 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb37460 0x55fe0cb378d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.478+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/916326454 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb35f10 0x55fe0cb36320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.478+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/916326454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb36860 0x55fe0cb36e80 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.478+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/916326454 >> 172.21.15.88:0/916326454 conn(0x55fe0cb2b630 msgr2=0x55fe0cb30980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.478+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/916326454 shutdown_connections 2023-07-14T17:44:29.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.478+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/916326454 wait complete. 2023-07-14T17:44:29.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.479+0000 7f0d0eea7b80 1 -- start start 2023-07-14T17:44:29.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.479+0000 7f0d0eea7b80 1 Processor -- start 2023-07-14T17:44:29.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.479+0000 7f0d0eea7b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb35f10 0x55fe0cb3f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.472 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.479+0000 7f0d0eea7b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb36860 0x55fe0cb3fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.479+0000 7f0d0eea7b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb37460 0x55fe0cb46e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.479+0000 7f0d0eea7b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe0ca26a70 con 0x55fe0cb35f10 2023-07-14T17:44:29.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.479+0000 7f0d0eea7b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe0ca26770 con 0x55fe0cb36860 2023-07-14T17:44:29.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.479+0000 7f0d0eea7b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe0c90a8d0 con 0x55fe0cb37460 2023-07-14T17:44:29.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.479+0000 7f0d0eea7b80 10 client.0 _ll_register_callbacks cb 0x55fe0ca24710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:29.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.479+0000 7f0d00aab700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb35f10 0x55fe0cb3f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.473 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.479+0000 7f0d00aab700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb35f10 0x55fe0cb3f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43320/0 (socket says 172.21.15.88:43320) 2023-07-14T17:44:29.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.479+0000 7f0d00aab700 1 -- 172.21.15.88:0/1572979560 learned_addr learned my addr 172.21.15.88:0/1572979560 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.479+0000 7f0cfbfff700 1 --2- 172.21.15.88:0/1572979560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb36860 0x55fe0cb3fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.479+0000 7f0d012ac700 1 --2- 172.21.15.88:0/1572979560 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb37460 0x55fe0cb46e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0d00aab700 1 -- 172.21.15.88:0/1572979560 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb37460 msgr2=0x55fe0cb46e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0d00aab700 1 --2- 172.21.15.88:0/1572979560 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb37460 0x55fe0cb46e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.474 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0d00aab700 1 -- 172.21.15.88:0/1572979560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb36860 msgr2=0x55fe0cb3fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0d00aab700 1 --2- 172.21.15.88:0/1572979560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb36860 0x55fe0cb3fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0d00aab700 1 -- 172.21.15.88:0/1572979560 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cec0099e0 con 0x55fe0cb35f10 2023-07-14T17:44:29.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0d00aab700 1 --2- 172.21.15.88:0/1572979560 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb35f10 0x55fe0cb3f8c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009990 tx=0x7f0ce401a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33a92f80826ccd8 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0cdbfff700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:29.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0cdbfff700 1 -- 172.21.15.88:0/1572979560 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce400bd20 con 0x55fe0cb35f10 2023-07-14T17:44:29.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0cdbfff700 1 -- 172.21.15.88:0/1572979560 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0ce400be80 con 0x55fe0cb35f10 2023-07-14T17:44:29.475 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/1572979560 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fe0c9e04a0 con 0x55fe0cb35f10 2023-07-14T17:44:29.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0cdbfff700 1 -- 172.21.15.88:0/1572979560 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ce4004500 con 0x55fe0cb35f10 2023-07-14T17:44:29.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0d0eea7b80 10 client.15495 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:29.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/1572979560 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55fe0c89dc10 con 0x55fe0cb35f10 2023-07-14T17:44:29.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0cdbfff700 1 -- 172.21.15.88:0/1572979560 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55fe0c9e04a0 con 0x55fe0cb35f10 2023-07-14T17:44:29.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0cdbfff700 1 -- 172.21.15.88:0/1572979560 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0ce400f580 con 0x55fe0cb35f10 2023-07-14T17:44:29.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0d0eea7b80 10 client.15495 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:29.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0d0eea7b80 10 client.15495 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T17:44:29.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0d0eea7b80 20 client.15495 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:29.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.480+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/1572979560 >> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] conn(0x55fe0cb4ab10 0x55fe0cb49f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.481+0000 7f0d0eea7b80 4 client.15495 mds_command: new command op to 24473 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T17:44:29.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.481+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/1572979560 --> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe0cb04d50 con 0x55fe0cb4ab10 2023-07-14T17:44:29.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.481+0000 7f0cfbfff700 1 --2- 172.21.15.88:0/1572979560 >> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] conn(0x55fe0cb4ab10 0x55fe0cb49f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.477 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.481+0000 7f0cdbfff700 10 client.15495 ms_handle_connect on v2:172.21.15.107:6836/1152608615 2023-07-14T17:44:29.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.481+0000 7f0cfbfff700 1 --2- 172.21.15.88:0/1572979560 >> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] conn(0x55fe0cb4ab10 0x55fe0cb49f40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d35fb1ca11a72bb2 server_cookie=5264370d1924ebde in_seq=0 out_seq=0 2023-07-14T17:44:29.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.482+0000 7f0cdbfff700 1 -- 172.21.15.88:0/1572979560 <== mds.1 v2:172.21.15.107:6836/1152608615 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55fe0cb04d50 con 0x55fe0cb4ab10 2023-07-14T17:44:29.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.482+0000 7f0cdbfff700 10 client.15495 handle_command_reply: tid=0 2023-07-14T17:44:29.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.482+0000 7f0d0eea7b80 1 client.15495 shutdown 2023-07-14T17:44:29.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.482+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/1572979560 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb35f10 msgr2=0x55fe0cb3f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.482+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/1572979560 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb35f10 0x55fe0cb3f8c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009990 tx=0x7f0ce401a050 comp rx=0 tx=0).stop 2023-07-14T17:44:29.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.482+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/1572979560 shutdown_connections 2023-07-14T17:44:29.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.482+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/1572979560 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb37460 0x55fe0cb46e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.482+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/1572979560 >> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] conn(0x55fe0cb4ab10 0x55fe0cb49f40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.482+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/1572979560 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb35f10 0x55fe0cb3f8c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.482+0000 7f0cdbfff700 0 client.15495 ms_handle_reset on v2:172.21.15.107:6836/1152608615 2023-07-14T17:44:29.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.482+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/1572979560 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb36860 0x55fe0cb3fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.482+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/1572979560 >> 172.21.15.88:0/1572979560 conn(0x55fe0cb2b630 msgr2=0x55fe0cb2f600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.482+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/1572979560 shutdown_connections 2023-07-14T17:44:29.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.482+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/1572979560 wait complete. 2023-07-14T17:44:29.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.482+0000 7f0d0eea7b80 20 client.15495 trim_cache size 0 max 16384 2023-07-14T17:44:29.481 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:29 smithi107 ceph-mon[128999]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2023-07-14T17:44:29.481 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:29 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2917064616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:44:29.481 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:29 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2110490728' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:29.481 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:29 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1271041221' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:29.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0d0eea7b80 1 Processor -- start 2023-07-14T17:44:29.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0d0eea7b80 1 -- start start 2023-07-14T17:44:29.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0d0eea7b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb35f10 0x55fe0cb300a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.483 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0d0eea7b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb36860 0x55fe0cb46e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0d0eea7b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb37460 0x55fe0cb49f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0d0eea7b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe0ca45480 con 0x55fe0cb36860 2023-07-14T17:44:29.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0d0eea7b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe0ca22d50 con 0x55fe0cb35f10 2023-07-14T17:44:29.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0d0eea7b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe0cb372e0 con 0x55fe0cb37460 2023-07-14T17:44:29.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0cfbfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb36860 0x55fe0cb46e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.484 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0cfbfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb36860 0x55fe0cb46e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43330/0 (socket says 172.21.15.88:43330) 2023-07-14T17:44:29.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0cfbfff700 1 -- 172.21.15.88:0/2919154219 learned_addr learned my addr 172.21.15.88:0/2919154219 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0d012ac700 1 --2- 172.21.15.88:0/2919154219 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb37460 0x55fe0cb49f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0cfbfff700 1 -- 172.21.15.88:0/2919154219 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb37460 msgr2=0x55fe0cb49f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0cfbfff700 1 --2- 172.21.15.88:0/2919154219 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb37460 0x55fe0cb49f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0cfbfff700 1 -- 172.21.15.88:0/2919154219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb35f10 msgr2=0x55fe0cb300a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0cfbfff700 1 --2- 172.21.15.88:0/2919154219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb35f10 0x55fe0cb300a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.485 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0cfbfff700 1 -- 172.21.15.88:0/2919154219 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe0c8db200 con 0x55fe0cb36860 2023-07-14T17:44:29.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0d012ac700 1 --2- 172.21.15.88:0/2919154219 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb37460 0x55fe0cb49f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:29.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.500+0000 7f0cfbfff700 1 --2- 172.21.15.88:0/2919154219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb36860 0x55fe0cb46e00 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0cec014040 tx=0x7f0cec005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ddc378c9215f883 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.501+0000 7f0cf37fe700 1 -- 172.21.15.88:0/2919154219 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec029070 con 0x55fe0cb36860 2023-07-14T17:44:29.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.501+0000 7f0cf37fe700 1 -- 172.21.15.88:0/2919154219 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0cec004100 con 0x55fe0cb36860 2023-07-14T17:44:29.486 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.501+0000 7f0cf37fe700 1 -- 172.21.15.88:0/2919154219 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec003680 con 0x55fe0cb36860 2023-07-14T17:44:29.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.501+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/2919154219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb36860 msgr2=0x55fe0cb46e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.501+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/2919154219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb36860 0x55fe0cb46e00 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0cec014040 tx=0x7f0cec005fa0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.491 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.501+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/2919154219 shutdown_connections 2023-07-14T17:44:29.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.501+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/2919154219 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb37460 0x55fe0cb49f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.501+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/2919154219 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb35f10 0x55fe0cb300a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.492 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.501+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/2919154219 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb36860 0x55fe0cb46e00 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.501+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/2919154219 >> 172.21.15.88:0/2919154219 conn(0x55fe0cb2b630 msgr2=0x55fe0cb30950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.501+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/2919154219 shutdown_connections 2023-07-14T17:44:29.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.501+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/2919154219 wait complete. 2023-07-14T17:44:29.493 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d0eea7b80 1 -- start start 2023-07-14T17:44:29.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d0eea7b80 1 Processor -- start 2023-07-14T17:44:29.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d0eea7b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb35f10 0x55fe0cb42810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.494 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d0eea7b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb36860 0x55fe0cb42d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d0eea7b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb37460 0x55fe0cb45d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d0eea7b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe0cb462b0 con 0x55fe0cb37460 2023-07-14T17:44:29.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d0eea7b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe0cb46420 con 0x55fe0cb36860 2023-07-14T17:44:29.495 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d0eea7b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe0cb465e0 con 0x55fe0cb35f10 2023-07-14T17:44:29.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d0eea7b80 10 client.0 _ll_register_callbacks cb 0x7f0ce4003680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:29.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d012ac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb37460 0x55fe0cb45d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d012ac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb37460 0x55fe0cb45d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43338/0 (socket says 172.21.15.88:43338) 2023-07-14T17:44:29.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d012ac700 1 -- 172.21.15.88:0/3973716111 learned_addr learned my addr 172.21.15.88:0/3973716111 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.496 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d00aab700 1 --2- 172.21.15.88:0/3973716111 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb35f10 0x55fe0cb42810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0cfbfff700 1 --2- 172.21.15.88:0/3973716111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb36860 0x55fe0cb42d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d012ac700 1 -- 172.21.15.88:0/3973716111 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb35f10 msgr2=0x55fe0cb42810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d012ac700 1 --2- 172.21.15.88:0/3973716111 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb35f10 0x55fe0cb42810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d012ac700 1 -- 172.21.15.88:0/3973716111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb36860 msgr2=0x55fe0cb42d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d012ac700 1 --2- 172.21.15.88:0/3973716111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb36860 0x55fe0cb42d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d012ac700 1 -- 172.21.15.88:0/3973716111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe0c8b4d90 con 0x55fe0cb37460 2023-07-14T17:44:29.497 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0d00aab700 1 --2- 172.21.15.88:0/3973716111 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb35f10 0x55fe0cb42810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:29.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.502+0000 7f0cfbfff700 1 --2- 172.21.15.88:0/3973716111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb36860 0x55fe0cb42d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:29.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.503+0000 7f0d012ac700 1 --2- 172.21.15.88:0/3973716111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb37460 0x55fe0cb45d70 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0cf40993e0 tx=0x7f0cf4099040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df89d498a5a53503 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.503+0000 7f0cdbfff700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:29.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.503+0000 7f0cdbfff700 1 -- 172.21.15.88:0/3973716111 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf400ddd0 con 0x55fe0cb37460 2023-07-14T17:44:29.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.503+0000 7f0cdbfff700 1 -- 172.21.15.88:0/3973716111 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0cf4099450 con 0x55fe0cb37460 2023-07-14T17:44:29.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.503+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/3973716111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0cec004390 con 0x55fe0cb37460 2023-07-14T17:44:29.498 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.503+0000 7f0cdbfff700 1 -- 172.21.15.88:0/3973716111 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0cf409d480 con 0x55fe0cb37460 2023-07-14T17:44:29.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.503+0000 7f0d0eea7b80 10 client.15501 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:29.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.503+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/3973716111 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55fe0c8b3260 con 0x55fe0cb37460 2023-07-14T17:44:29.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.503+0000 7f0cdbfff700 1 -- 172.21.15.88:0/3973716111 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0cf40b5080 con 0x55fe0cb37460 2023-07-14T17:44:29.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.503+0000 7f0cdbfff700 1 -- 172.21.15.88:0/3973716111 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0cf4098940 con 0x55fe0cb37460 2023-07-14T17:44:29.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.503+0000 7f0d0eea7b80 10 client.15501 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:29.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.503+0000 7f0d0eea7b80 10 client.15501 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T17:44:29.499 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.503+0000 7f0d0eea7b80 20 client.15501 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:29.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.503+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/3973716111 >> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] conn(0x55fe0cba8ac0 0x55fe0cba5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.503+0000 7f0d0eea7b80 4 client.15501 mds_command: new command op to 24473 tid=0[{"prefix": "status"}] 2023-07-14T17:44:29.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.503+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/3973716111 --> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fe0ca7fe60 con 0x55fe0cba8ac0 2023-07-14T17:44:29.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.503+0000 7f0d00aab700 1 --2- 172.21.15.88:0/3973716111 >> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] conn(0x55fe0cba8ac0 0x55fe0cba5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.504+0000 7f0cdbfff700 10 client.15501 ms_handle_connect on v2:172.21.15.107:6836/1152608615 2023-07-14T17:44:29.500 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.504+0000 7f0d00aab700 1 --2- 172.21.15.88:0/3973716111 >> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] conn(0x55fe0cba8ac0 0x55fe0cba5430 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f9de9da7250a1b5c server_cookie=25eb238ac928cfba in_seq=0 out_seq=0 2023-07-14T17:44:29.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.504+0000 7f0cdbfff700 1 -- 172.21.15.88:0/3973716111 <== mds.1 v2:172.21.15.107:6836/1152608615 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55fe0ca7fe60 con 0x55fe0cba8ac0 2023-07-14T17:44:29.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.504+0000 7f0cdbfff700 10 client.15501 handle_command_reply: tid=0 2023-07-14T17:44:29.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.504+0000 7f0d0eea7b80 1 client.15501 shutdown 2023-07-14T17:44:29.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.504+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/3973716111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb37460 msgr2=0x55fe0cb45d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.504+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/3973716111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb37460 0x55fe0cb45d70 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f0cf40993e0 tx=0x7f0cf4099040 comp rx=0 tx=0).stop 2023-07-14T17:44:29.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.504+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/3973716111 shutdown_connections 2023-07-14T17:44:29.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.504+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/3973716111 >> [v2:172.21.15.107:6836/1152608615,v1:172.21.15.107:6837/1152608615] conn(0x55fe0cba8ac0 0x55fe0cba5430 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.504+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/3973716111 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55fe0cb37460 0x55fe0cb45d70 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.504+0000 7f0cdbfff700 0 client.15501 ms_handle_reset on v2:172.21.15.107:6836/1152608615 2023-07-14T17:44:29.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.504+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/3973716111 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x55fe0cb35f10 0x55fe0cb42810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.504+0000 7f0d0eea7b80 1 --2- 172.21.15.88:0/3973716111 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x55fe0cb36860 0x55fe0cb42d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.504+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/3973716111 >> 172.21.15.88:0/3973716111 conn(0x55fe0cb2b630 msgr2=0x55fe0cb3d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.504+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/3973716111 shutdown_connections 2023-07-14T17:44:29.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.504+0000 7f0d0eea7b80 1 -- 172.21.15.88:0/3973716111 wait complete. 2023-07-14T17:44:29.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.504+0000 7f0d0eea7b80 20 client.15501 trim_cache size 0 max 16384 2023-07-14T17:44:29.503 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-07-14T17:44:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "b462a556-226c-11ee-9b33-001a4aab830c", 2023-07-14T17:44:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 1, 2023-07-14T17:44:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 24473, 2023-07-14T17:44:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2023-07-14T17:44:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2023-07-14T17:44:29.504 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2023-07-14T17:44:29.505 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 88.772301662999993, 2023-07-14T17:44:29.505 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 27, 2023-07-14T17:44:29.505 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 88, 2023-07-14T17:44:29.505 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 86, 2023-07-14T17:44:29.505 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 120.77103721100001 2023-07-14T17:44:29.505 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-07-14T17:44:29.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.505+0000 7f0d02d0f700 1 -- 172.21.15.88:0/148568654 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0ce80a1150 msgr2=0x7f0ce80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.505+0000 7f0d02d0f700 1 --2- 172.21.15.88:0/148568654 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0ce80a1150 0x7f0ce80a3600 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0ce40098e0 tx=0x7f0ce400b260 comp rx=0 tx=0).stop 2023-07-14T17:44:29.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.505+0000 7f0d02d0f700 1 -- 172.21.15.88:0/148568654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cfc100690 msgr2=0x7f0cfc19bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.505+0000 7f0d02d0f700 1 --2- 172.21.15.88:0/148568654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cfc100690 0x7f0cfc19bde0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0cf4007ee0 tx=0x7f0cf400d3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.505+0000 7f0d02d0f700 1 -- 172.21.15.88:0/148568654 shutdown_connections 2023-07-14T17:44:29.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.505+0000 7f0d02d0f700 1 --2- 172.21.15.88:0/148568654 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0ce80a1150 0x7f0ce80a3600 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.505+0000 7f0d02d0f700 1 --2- 172.21.15.88:0/148568654 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0cfc100690 0x7f0cfc19bde0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.505+0000 7f0d02d0f700 1 --2- 172.21.15.88:0/148568654 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0cfc0feb10 0x7f0cfc194830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.505+0000 7f0d02d0f700 1 --2- 172.21.15.88:0/148568654 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0cfc0ffd00 0x7f0cfc194d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.505+0000 7f0d02d0f700 1 -- 172.21.15.88:0/148568654 >> 172.21.15.88:0/148568654 conn(0x7f0cfc0fa130 msgr2=0x7f0cfc0fc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.505+0000 7f0d02d0f700 1 -- 172.21.15.88:0/148568654 shutdown_connections 2023-07-14T17:44:29.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.505+0000 7f0d02d0f700 1 -- 172.21.15.88:0/148568654 wait complete. 2023-07-14T17:44:29.509 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-14T17:44:29.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.648+0000 7f959a40d700 1 -- 172.21.15.88:0/3308487396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95940ff470 msgr2=0x7f95940ff8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.648+0000 7f959a40d700 1 --2- 172.21.15.88:0/3308487396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95940ff470 0x7f95940ff8c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9588009990 tx=0x7f9588009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.648+0000 7f959a40d700 1 -- 172.21.15.88:0/3308487396 shutdown_connections 2023-07-14T17:44:29.633 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.648+0000 7f959a40d700 1 --2- 172.21.15.88:0/3308487396 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95940ffe00 0x7f95941066c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.648+0000 7f959a40d700 1 --2- 172.21.15.88:0/3308487396 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95940fe280 0x7f95940fe690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.648+0000 7f959a40d700 1 --2- 172.21.15.88:0/3308487396 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95940ff470 0x7f95940ff8c0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.648+0000 7f959a40d700 1 -- 172.21.15.88:0/3308487396 >> 172.21.15.88:0/3308487396 conn(0x7f95940f98a0 msgr2=0x7f95940fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.648+0000 7f959a40d700 1 -- 172.21.15.88:0/3308487396 shutdown_connections 2023-07-14T17:44:29.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.648+0000 7f959a40d700 1 -- 172.21.15.88:0/3308487396 wait complete. 2023-07-14T17:44:29.634 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.649+0000 7f959a40d700 1 Processor -- start 2023-07-14T17:44:29.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.649+0000 7f959a40d700 1 -- start start 2023-07-14T17:44:29.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.649+0000 7f959a40d700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95940fe280 0x7f9594107b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.649+0000 7f959a40d700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95940ff470 0x7f959419a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.649+0000 7f959a40d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95940ffe00 0x7f959419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.649+0000 7f959a40d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959419d410 con 0x7f95940ffe00 2023-07-14T17:44:29.635 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.649+0000 7f959a40d700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959419d580 con 0x7f95940ff470 2023-07-14T17:44:29.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.649+0000 7f959a40d700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959419d740 con 0x7f95940fe280 2023-07-14T17:44:29.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.649+0000 7f95989aa700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95940ffe00 0x7f959419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.649+0000 7f95989aa700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95940ffe00 0x7f959419cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43350/0 (socket says 172.21.15.88:43350) 2023-07-14T17:44:29.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.649+0000 7f95989aa700 1 -- 172.21.15.88:0/1070964392 learned_addr learned my addr 172.21.15.88:0/1070964392 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.650+0000 7f9593fff700 1 --2- 172.21.15.88:0/1070964392 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95940fe280 0x7f9594107b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.636 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.650+0000 7f95937fe700 1 --2- 172.21.15.88:0/1070964392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95940ff470 0x7f959419a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.650+0000 7f95989aa700 1 -- 172.21.15.88:0/1070964392 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95940fe280 msgr2=0x7f9594107b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.650+0000 7f95989aa700 1 --2- 172.21.15.88:0/1070964392 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95940fe280 0x7f9594107b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.650+0000 7f95989aa700 1 -- 172.21.15.88:0/1070964392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95940ff470 msgr2=0x7f959419a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.650+0000 7f95989aa700 1 --2- 172.21.15.88:0/1070964392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95940ff470 0x7f959419a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.637 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.650+0000 7f95989aa700 1 -- 172.21.15.88:0/1070964392 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95880095f0 con 0x7f95940ffe00 2023-07-14T17:44:29.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.650+0000 7f95937fe700 1 --2- 172.21.15.88:0/1070964392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95940ff470 0x7f959419a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:29.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.650+0000 7f95989aa700 1 --2- 172.21.15.88:0/1070964392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95940ffe00 0x7f959419cc40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f958c007ee0 tx=0x7f958c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.638 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.650+0000 7f95917fa700 1 -- 172.21.15.88:0/1070964392 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958c00ddd0 con 0x7f95940ffe00 2023-07-14T17:44:29.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.650+0000 7f95917fa700 1 -- 172.21.15.88:0/1070964392 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f958c00f040 con 0x7f95940ffe00 2023-07-14T17:44:29.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.650+0000 7f959a40d700 1 -- 172.21.15.88:0/1070964392 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f959419da20 con 0x7f95940ffe00 2023-07-14T17:44:29.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.650+0000 7f95917fa700 1 -- 172.21.15.88:0/1070964392 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958c0047d0 con 0x7f95940ffe00 2023-07-14T17:44:29.639 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.650+0000 7f959a40d700 1 -- 172.21.15.88:0/1070964392 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f959419df30 con 0x7f95940ffe00 2023-07-14T17:44:29.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.652+0000 7f95917fa700 1 -- 172.21.15.88:0/1070964392 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f958c003a00 con 0x7f95940ffe00 2023-07-14T17:44:29.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.653+0000 7f95917fa700 1 --2- 172.21.15.88:0/1070964392 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f957c0a1150 0x7f957c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.647 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.653+0000 7f9593fff700 1 --2- 172.21.15.88:0/1070964392 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f957c0a1150 0x7f957c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.653+0000 7f95917fa700 1 -- 172.21.15.88:0/1070964392 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f958c0cc4d0 con 0x7f95940ffe00 2023-07-14T17:44:29.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.654+0000 7f9593fff700 1 --2- 172.21.15.88:0/1070964392 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f957c0a1150 0x7f957c0a3600 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f95940ff2d0 tx=0x7f9584009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.654+0000 7f95a65a5b80 1 Processor -- start 2023-07-14T17:44:29.648 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.654+0000 7f95a65a5b80 1 -- start start 2023-07-14T17:44:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.654+0000 7f95a65a5b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9baf10 0x5598bc9bb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.654+0000 7f95a65a5b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9bb860 0x5598bc9bbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.654+0000 7f95a65a5b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9bc460 0x5598bc9bc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.654+0000 7f95a65a5b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598bc8bfc60 con 0x5598bc9baf10 2023-07-14T17:44:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.654+0000 7f95a65a5b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598bc9129f0 con 0x5598bc9bc460 2023-07-14T17:44:29.649 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.654+0000 7f95a65a5b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598bc90be80 con 0x5598bc9bb860 2023-07-14T17:44:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.654+0000 7f9593fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9baf10 0x5598bc9bb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.654+0000 7f9593fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9baf10 0x5598bc9bb320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43354/0 (socket says 172.21.15.88:43354) 2023-07-14T17:44:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.654+0000 7f9593fff700 1 -- 172.21.15.88:0/1693447163 learned_addr learned my addr 172.21.15.88:0/1693447163 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.654+0000 7f95989aa700 1 --2- 172.21.15.88:0/1693447163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9bc460 0x5598bc9bc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.654+0000 7f95937fe700 1 --2- 172.21.15.88:0/1693447163 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9bb860 0x5598bc9bbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.654+0000 7f9593fff700 1 -- 172.21.15.88:0/1693447163 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9bb860 msgr2=0x5598bc9bbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.654+0000 7f9593fff700 1 --2- 172.21.15.88:0/1693447163 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9bb860 0x5598bc9bbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.654+0000 7f9593fff700 1 -- 172.21.15.88:0/1693447163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9bc460 msgr2=0x5598bc9bc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.654+0000 7f9593fff700 1 --2- 172.21.15.88:0/1693447163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9bc460 0x5598bc9bc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.655+0000 7f9593fff700 1 -- 172.21.15.88:0/1693447163 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598bc739d90 con 0x5598bc9baf10 2023-07-14T17:44:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.655+0000 7f95937fe700 1 --2- 172.21.15.88:0/1693447163 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9bb860 0x5598bc9bbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:29.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.655+0000 7f9593fff700 1 --2- 172.21.15.88:0/1693447163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9baf10 0x5598bc9bb320 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9584011d50 tx=0x7f9584014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e370b7cfeca80be9 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.655+0000 7f957a7fc700 1 -- 172.21.15.88:0/1693447163 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958400edd0 con 0x5598bc9baf10 2023-07-14T17:44:29.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.655+0000 7f957a7fc700 1 -- 172.21.15.88:0/1693447163 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9584021070 con 0x5598bc9baf10 2023-07-14T17:44:29.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.655+0000 7f957a7fc700 1 -- 172.21.15.88:0/1693447163 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958401d670 con 0x5598bc9baf10 2023-07-14T17:44:29.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.655+0000 7f95a65a5b80 1 -- 172.21.15.88:0/1693447163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9baf10 msgr2=0x5598bc9bb320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.652 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.655+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/1693447163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9baf10 0x5598bc9bb320 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f9584011d50 tx=0x7f9584014000 comp rx=0 tx=0).stop 2023-07-14T17:44:29.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.655+0000 7f95a65a5b80 1 -- 172.21.15.88:0/1693447163 shutdown_connections 2023-07-14T17:44:29.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.655+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/1693447163 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9bc460 0x5598bc9bc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.655+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/1693447163 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9baf10 0x5598bc9bb320 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.656+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/1693447163 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9bb860 0x5598bc9bbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.656+0000 7f95a65a5b80 1 -- 172.21.15.88:0/1693447163 >> 172.21.15.88:0/1693447163 conn(0x5598bc9b0630 msgr2=0x5598bc9b5980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.656+0000 7f95a65a5b80 1 -- 172.21.15.88:0/1693447163 shutdown_connections 2023-07-14T17:44:29.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.656+0000 7f95a65a5b80 1 -- 172.21.15.88:0/1693447163 wait complete. 2023-07-14T17:44:29.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.656+0000 7f95a65a5b80 1 -- start start 2023-07-14T17:44:29.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.656+0000 7f95a65a5b80 1 Processor -- start 2023-07-14T17:44:29.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.656+0000 7f95a65a5b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9baf10 0x5598bc9c48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.656+0000 7f95a65a5b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9bb860 0x5598bc9c4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.656+0000 7f95a65a5b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9bc460 0x5598bc9cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.656+0000 7f95a65a5b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598bc8aba70 con 0x5598bc9bb860 2023-07-14T17:44:29.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.656+0000 7f95a65a5b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598bc8ab770 con 0x5598bc9bc460 2023-07-14T17:44:29.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.656+0000 7f95a65a5b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598bc78f8d0 con 0x5598bc9baf10 2023-07-14T17:44:29.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.656+0000 7f95a65a5b80 10 client.0 _ll_register_callbacks cb 0x5598bc8a9710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:29.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.656+0000 7f95937fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9bb860 0x5598bc9c4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.656+0000 7f95937fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9bb860 0x5598bc9c4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43362/0 (socket says 172.21.15.88:43362) 2023-07-14T17:44:29.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.657+0000 7f95937fe700 1 -- 172.21.15.88:0/2997910292 learned_addr learned my addr 172.21.15.88:0/2997910292 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.656 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.657+0000 7f95989aa700 1 --2- 172.21.15.88:0/2997910292 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9bc460 0x5598bc9cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.657+0000 7f9593fff700 1 --2- 172.21.15.88:0/2997910292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9baf10 0x5598bc9c48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.657+0000 7f95937fe700 1 -- 172.21.15.88:0/2997910292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9baf10 msgr2=0x5598bc9c48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.657+0000 7f95937fe700 1 --2- 172.21.15.88:0/2997910292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9baf10 0x5598bc9c48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.657+0000 7f95937fe700 1 -- 172.21.15.88:0/2997910292 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9bc460 msgr2=0x5598bc9cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.657+0000 7f95937fe700 1 --2- 172.21.15.88:0/2997910292 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9bc460 0x5598bc9cbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.657+0000 7f95937fe700 1 -- 172.21.15.88:0/2997910292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9584017070 con 0x5598bc9bb860 2023-07-14T17:44:29.657 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.657+0000 7f9593fff700 1 --2- 172.21.15.88:0/2997910292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9baf10 0x5598bc9c48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:29.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.657+0000 7f95937fe700 1 --2- 172.21.15.88:0/2997910292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9bb860 0x5598bc9c4e00 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9588003f60 tx=0x7f9588003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ade3068172a770 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.657+0000 7f95737fe700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:29.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.657+0000 7f95737fe700 1 -- 172.21.15.88:0/2997910292 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958800b8b0 con 0x5598bc9bb860 2023-07-14T17:44:29.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.657+0000 7f95a65a5b80 1 -- 172.21.15.88:0/2997910292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5598bc8654a0 con 0x5598bc9bb860 2023-07-14T17:44:29.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.657+0000 7f95737fe700 1 -- 172.21.15.88:0/2997910292 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f958800ba10 con 0x5598bc9bb860 2023-07-14T17:44:29.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.657+0000 7f95737fe700 1 -- 172.21.15.88:0/2997910292 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f958800bbd0 con 0x5598bc9bb860 2023-07-14T17:44:29.658 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.657+0000 7f95a65a5b80 10 client.15513 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:29.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.657+0000 7f95a65a5b80 1 -- 172.21.15.88:0/2997910292 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5598bc722c10 con 0x5598bc9bb860 2023-07-14T17:44:29.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.657+0000 7f95737fe700 1 -- 172.21.15.88:0/2997910292 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x5598bc8654a0 con 0x5598bc9bb860 2023-07-14T17:44:29.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.658+0000 7f95737fe700 1 -- 172.21.15.88:0/2997910292 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f95880173f0 con 0x5598bc9bb860 2023-07-14T17:44:29.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.658+0000 7f95a65a5b80 10 client.15513 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:29.659 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.658+0000 7f95a65a5b80 10 client.15513 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T17:44:29.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.658+0000 7f95a65a5b80 20 client.15513 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:29.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.658+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/2997910292 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x5598bc9cfad0 0x5598bc9cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.658+0000 7f95a65a5b80 4 client.15513 mds_command: new command op to 24506 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T17:44:29.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.658+0000 7f95a65a5b80 1 -- 172.21.15.88:0/2997910292 --> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5598bc989d50 con 0x5598bc9cfad0 2023-07-14T17:44:29.660 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.658+0000 7f9593fff700 1 --2- 172.21.15.88:0/2997910292 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x5598bc9cfad0 0x5598bc9cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.658+0000 7f95737fe700 10 client.15513 ms_handle_connect on v2:172.21.15.107:6838/31646548 2023-07-14T17:44:29.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.658+0000 7f9593fff700 1 --2- 172.21.15.88:0/2997910292 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x5598bc9cfad0 0x5598bc9cef00 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b2cd7b48d5877ac server_cookie=53f8a786039b5b77 in_seq=0 out_seq=0 2023-07-14T17:44:29.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.659+0000 7f95737fe700 1 -- 172.21.15.88:0/2997910292 <== mds.0 v2:172.21.15.107:6838/31646548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x5598bc989d50 con 0x5598bc9cfad0 2023-07-14T17:44:29.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.659+0000 7f95737fe700 10 client.15513 handle_command_reply: tid=0 2023-07-14T17:44:29.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.659+0000 7f95a65a5b80 1 client.15513 shutdown 2023-07-14T17:44:29.661 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.659+0000 7f95a65a5b80 1 -- 172.21.15.88:0/2997910292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9bb860 msgr2=0x5598bc9c4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.659+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/2997910292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9bb860 0x5598bc9c4e00 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9588003f60 tx=0x7f9588003680 comp rx=0 tx=0).stop 2023-07-14T17:44:29.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.659+0000 7f95a65a5b80 1 -- 172.21.15.88:0/2997910292 shutdown_connections 2023-07-14T17:44:29.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.659+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/2997910292 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x5598bc9cfad0 0x5598bc9cef00 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.659+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/2997910292 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9bc460 0x5598bc9cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.662 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.659+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/2997910292 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9baf10 0x5598bc9c48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.659+0000 7f95737fe700 0 client.15513 ms_handle_reset on v2:172.21.15.107:6838/31646548 2023-07-14T17:44:29.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.659+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/2997910292 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9bb860 0x5598bc9c4e00 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.659+0000 7f95a65a5b80 1 -- 172.21.15.88:0/2997910292 >> 172.21.15.88:0/2997910292 conn(0x5598bc9b0630 msgr2=0x5598bc9b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.659+0000 7f95a65a5b80 1 -- 172.21.15.88:0/2997910292 shutdown_connections 2023-07-14T17:44:29.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.659+0000 7f95a65a5b80 1 -- 172.21.15.88:0/2997910292 wait complete. 2023-07-14T17:44:29.663 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.659+0000 7f95a65a5b80 20 client.15513 trim_cache size 0 max 16384 2023-07-14T17:44:29.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95a65a5b80 1 Processor -- start 2023-07-14T17:44:29.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95a65a5b80 1 -- start start 2023-07-14T17:44:29.668 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95a65a5b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9baf10 0x5598bc9b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95a65a5b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9bb860 0x5598bc9cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95a65a5b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9bc460 0x5598bc9c27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95a65a5b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598bc9b7510 con 0x5598bc9bc460 2023-07-14T17:44:29.669 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95a65a5b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598bc9c5ab0 con 0x5598bc9baf10 2023-07-14T17:44:29.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95a65a5b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598bc8da790 con 0x5598bc9bb860 2023-07-14T17:44:29.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95989aa700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9bc460 0x5598bc9c27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95989aa700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9bc460 0x5598bc9c27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43372/0 (socket says 172.21.15.88:43372) 2023-07-14T17:44:29.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95989aa700 1 -- 172.21.15.88:0/115155740 learned_addr learned my addr 172.21.15.88:0/115155740 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.670 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95937fe700 1 --2- 172.21.15.88:0/115155740 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9bb860 0x5598bc9cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f9593fff700 1 --2- 172.21.15.88:0/115155740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9baf10 0x5598bc9b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95989aa700 1 -- 172.21.15.88:0/115155740 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9bb860 msgr2=0x5598bc9cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95989aa700 1 --2- 172.21.15.88:0/115155740 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9bb860 0x5598bc9cbe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95989aa700 1 -- 172.21.15.88:0/115155740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9baf10 msgr2=0x5598bc9b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95989aa700 1 --2- 172.21.15.88:0/115155740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9baf10 0x5598bc9b50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.671 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95989aa700 1 -- 172.21.15.88:0/115155740 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5598bc760200 con 0x5598bc9bc460 2023-07-14T17:44:29.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95937fe700 1 --2- 172.21.15.88:0/115155740 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9bb860 0x5598bc9cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:29.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.677+0000 7f95989aa700 1 --2- 172.21.15.88:0/115155740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9bc460 0x5598bc9c27f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f958c0992d0 tx=0x7f958c0b9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a50098a9004fe2d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.678+0000 7f957affd700 1 -- 172.21.15.88:0/115155740 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958c098f30 con 0x5598bc9bc460 2023-07-14T17:44:29.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.678+0000 7f957affd700 1 -- 172.21.15.88:0/115155740 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f958c0b90b0 con 0x5598bc9bc460 2023-07-14T17:44:29.672 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.678+0000 7f957affd700 1 -- 172.21.15.88:0/115155740 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958c003d40 con 0x5598bc9bc460 2023-07-14T17:44:29.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.678+0000 7f95a65a5b80 1 -- 172.21.15.88:0/115155740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9bc460 msgr2=0x5598bc9c27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.678+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/115155740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9bc460 0x5598bc9c27f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f958c0992d0 tx=0x7f958c0b9040 comp rx=0 tx=0).stop 2023-07-14T17:44:29.673 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.678+0000 7f95a65a5b80 1 -- 172.21.15.88:0/115155740 shutdown_connections 2023-07-14T17:44:29.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.678+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/115155740 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9bc460 0x5598bc9c27f0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.678+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/115155740 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9baf10 0x5598bc9b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.678+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/115155740 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9bb860 0x5598bc9cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.674 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.678+0000 7f95a65a5b80 1 -- 172.21.15.88:0/115155740 >> 172.21.15.88:0/115155740 conn(0x5598bc9b0630 msgr2=0x5598bc9b1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.678+0000 7f95a65a5b80 1 -- 172.21.15.88:0/115155740 shutdown_connections 2023-07-14T17:44:29.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.678+0000 7f95a65a5b80 1 -- 172.21.15.88:0/115155740 wait complete. 2023-07-14T17:44:29.675 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.679+0000 7f95a65a5b80 1 -- start start 2023-07-14T17:44:29.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.679+0000 7f95a65a5b80 1 Processor -- start 2023-07-14T17:44:29.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.679+0000 7f95a65a5b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9baf10 0x5598bc9c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.676 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.679+0000 7f95a65a5b80 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9bb860 0x5598bca21720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.679+0000 7f95a65a5b80 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9bc460 0x5598bca23c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.679+0000 7f95a65a5b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598bca241f0 con 0x5598bc9baf10 2023-07-14T17:44:29.677 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.679+0000 7f95a65a5b80 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598bca24360 con 0x5598bc9bb860 2023-07-14T17:44:29.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.679+0000 7f95a65a5b80 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598bca24520 con 0x5598bc9bc460 2023-07-14T17:44:29.678 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.679+0000 7f95a65a5b80 10 client.0 _ll_register_callbacks cb 0x7f9584005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T17:44:29.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.679+0000 7f9593fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9baf10 0x5598bc9c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.679+0000 7f9593fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9baf10 0x5598bc9c9df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43374/0 (socket says 172.21.15.88:43374) 2023-07-14T17:44:29.679 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.679+0000 7f9593fff700 1 -- 172.21.15.88:0/1007715845 learned_addr learned my addr 172.21.15.88:0/1007715845 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:29.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.679+0000 7f9593fff700 1 -- 172.21.15.88:0/1007715845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9bc460 msgr2=0x5598bca23c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.679+0000 7f9593fff700 1 --2- 172.21.15.88:0/1007715845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9bc460 0x5598bca23c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.679+0000 7f95937fe700 1 --2- 172.21.15.88:0/1007715845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9bb860 0x5598bca21720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.680 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.679+0000 7f9593fff700 1 -- 172.21.15.88:0/1007715845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9bb860 msgr2=0x5598bca21720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T17:44:29.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.679+0000 7f9593fff700 1 --2- 172.21.15.88:0/1007715845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9bb860 0x5598bca21720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.680+0000 7f9593fff700 1 -- 172.21.15.88:0/1007715845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f959419df30 con 0x5598bc9baf10 2023-07-14T17:44:29.681 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.680+0000 7f9593fff700 1 --2- 172.21.15.88:0/1007715845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9baf10 0x5598bc9c9df0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f958400ea20 tx=0x7f9584013ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209aeb5ab337fc67 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:29.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.680+0000 7f95737fe700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T17:44:29.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.680+0000 7f95737fe700 1 -- 172.21.15.88:0/1007715845 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9584033110 con 0x5598bc9baf10 2023-07-14T17:44:29.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.680+0000 7f95737fe700 1 -- 172.21.15.88:0/1007715845 <== mon.0 v2:172.21.15.88:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9584011e40 con 0x5598bc9baf10 2023-07-14T17:44:29.682 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.680+0000 7f95a65a5b80 1 -- 172.21.15.88:0/1007715845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9584021070 con 0x5598bc9baf10 2023-07-14T17:44:29.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.680+0000 7f95737fe700 1 -- 172.21.15.88:0/1007715845 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9584033270 con 0x5598bc9baf10 2023-07-14T17:44:29.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.680+0000 7f95a65a5b80 10 client.15519 fetch_fsmap learned FSMap version 27 2023-07-14T17:44:29.683 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.680+0000 7f95a65a5b80 1 -- 172.21.15.88:0/1007715845 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5598bc738260 con 0x5598bc9baf10 2023-07-14T17:44:29.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.680+0000 7f95737fe700 1 -- 172.21.15.88:0/1007715845 <== mon.0 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9584021070 con 0x5598bc9baf10 2023-07-14T17:44:29.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.680+0000 7f95737fe700 1 -- 172.21.15.88:0/1007715845 <== mon.0 v2:172.21.15.88:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f958403b070 con 0x5598bc9baf10 2023-07-14T17:44:29.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.680+0000 7f95a65a5b80 10 client.15519 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T17:44:29.684 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.680+0000 7f95a65a5b80 10 client.15519 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T17:44:29.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.680+0000 7f95a65a5b80 20 client.15519 populate_metadata read hostname 'smithi088' 2023-07-14T17:44:29.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.680+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/1007715845 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x5598bca293a0 0x5598bca25c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:29.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.680+0000 7f95a65a5b80 4 client.15519 mds_command: new command op to 24506 tid=0[{"prefix": "status"}] 2023-07-14T17:44:29.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.680+0000 7f95a65a5b80 1 -- 172.21.15.88:0/1007715845 --> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5598bc904e60 con 0x5598bca293a0 2023-07-14T17:44:29.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.681+0000 7f95937fe700 1 --2- 172.21.15.88:0/1007715845 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x5598bca293a0 0x5598bca25c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:29.685 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.681+0000 7f95737fe700 10 client.15519 ms_handle_connect on v2:172.21.15.107:6838/31646548 2023-07-14T17:44:29.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.681+0000 7f95937fe700 1 --2- 172.21.15.88:0/1007715845 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x5598bca293a0 0x5598bca25c90 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=715ddc73ccf2d210 server_cookie=41cc01a169cf25a0 in_seq=0 out_seq=0 2023-07-14T17:44:29.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.681+0000 7f95737fe700 1 -- 172.21.15.88:0/1007715845 <== mds.0 v2:172.21.15.107:6838/31646548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5598bc904e60 con 0x5598bca293a0 2023-07-14T17:44:29.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.681+0000 7f95737fe700 10 client.15519 handle_command_reply: tid=0 2023-07-14T17:44:29.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.681+0000 7f95a65a5b80 1 client.15519 shutdown 2023-07-14T17:44:29.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.682+0000 7f95a65a5b80 1 -- 172.21.15.88:0/1007715845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9baf10 msgr2=0x5598bc9c9df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.682+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/1007715845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9baf10 0x5598bc9c9df0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f958400ea20 tx=0x7f9584013ee0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.686 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.682+0000 7f95a65a5b80 1 -- 172.21.15.88:0/1007715845 shutdown_connections 2023-07-14T17:44:29.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.682+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/1007715845 >> [v2:172.21.15.107:6838/31646548,v1:172.21.15.107:6839/31646548] conn(0x5598bca293a0 0x5598bca25c90 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.682+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/1007715845 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x5598bc9bc460 0x5598bca23c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.682+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/1007715845 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5598bc9baf10 0x5598bc9c9df0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.682+0000 7f95a65a5b80 1 --2- 172.21.15.88:0/1007715845 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x5598bc9bb860 0x5598bca21720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.682+0000 7f95737fe700 0 client.15519 ms_handle_reset on v2:172.21.15.107:6838/31646548 2023-07-14T17:44:29.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.682+0000 7f95a65a5b80 1 -- 172.21.15.88:0/1007715845 >> 172.21.15.88:0/1007715845 conn(0x5598bc9b0630 msgr2=0x5598bc9b50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.687 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.682+0000 7f95a65a5b80 1 -- 172.21.15.88:0/1007715845 shutdown_connections 2023-07-14T17:44:29.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.682+0000 7f95a65a5b80 1 -- 172.21.15.88:0/1007715845 wait complete. 2023-07-14T17:44:29.688 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.682+0000 7f95a65a5b80 20 client.15519 trim_cache size 0 max 16384 2023-07-14T17:44:29.689 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-07-14T17:44:29.689 INFO:teuthology.orchestra.run.smithi088.stdout: "cluster_fsid": "b462a556-226c-11ee-9b33-001a4aab830c", 2023-07-14T17:44:29.689 INFO:teuthology.orchestra.run.smithi088.stdout: "whoami": 0, 2023-07-14T17:44:29.689 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 24506, 2023-07-14T17:44:29.689 INFO:teuthology.orchestra.run.smithi088.stdout: "want_state": "up:active", 2023-07-14T17:44:29.690 INFO:teuthology.orchestra.run.smithi088.stdout: "state": "up:active", 2023-07-14T17:44:29.690 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs", 2023-07-14T17:44:29.690 INFO:teuthology.orchestra.run.smithi088.stdout: "rank_uptime": 106.767547065, 2023-07-14T17:44:29.690 INFO:teuthology.orchestra.run.smithi088.stdout: "mdsmap_epoch": 27, 2023-07-14T17:44:29.690 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch": 88, 2023-07-14T17:44:29.690 INFO:teuthology.orchestra.run.smithi088.stdout: "osdmap_epoch_barrier": 86, 2023-07-14T17:44:29.691 INFO:teuthology.orchestra.run.smithi088.stdout: "uptime": 111.85389103200001 2023-07-14T17:44:29.691 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-07-14T17:44:29.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.682+0000 7f959a40d700 1 -- 172.21.15.88:0/1070964392 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f957c0a1150 msgr2=0x7f957c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.682+0000 7f959a40d700 1 --2- 172.21.15.88:0/1070964392 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f957c0a1150 0x7f957c0a3600 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f95940ff2d0 tx=0x7f9584009260 comp rx=0 tx=0).stop 2023-07-14T17:44:29.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.682+0000 7f959a40d700 1 -- 172.21.15.88:0/1070964392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95940ffe00 msgr2=0x7f959419cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:29.692 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.682+0000 7f959a40d700 1 --2- 172.21.15.88:0/1070964392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95940ffe00 0x7f959419cc40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f958c007ee0 tx=0x7f958c00d3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.683+0000 7f959a40d700 1 -- 172.21.15.88:0/1070964392 shutdown_connections 2023-07-14T17:44:29.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.683+0000 7f959a40d700 1 --2- 172.21.15.88:0/1070964392 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f957c0a1150 0x7f957c0a3600 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.683+0000 7f959a40d700 1 --2- 172.21.15.88:0/1070964392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95940ffe00 0x7f959419cc40 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.683+0000 7f959a40d700 1 --2- 172.21.15.88:0/1070964392 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f95940fe280 0x7f9594107b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.683+0000 7f959a40d700 1 --2- 172.21.15.88:0/1070964392 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f95940ff470 0x7f959419a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:29.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.683+0000 7f959a40d700 1 -- 172.21.15.88:0/1070964392 >> 172.21.15.88:0/1070964392 conn(0x7f95940f98a0 msgr2=0x7f95940fbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:29.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.683+0000 7f959a40d700 1 -- 172.21.15.88:0/1070964392 shutdown_connections 2023-07-14T17:44:29.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:29.683+0000 7f959a40d700 1 -- 172.21.15.88:0/1070964392 wait complete. 2023-07-14T17:44:29.694 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-07-14T17:44:29.710 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-07-14T17:44:29.710 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2023-07-14T17:44:29.711 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2023-07-14T17:44:29.711 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-07-14T17:44:29.711 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-14T17:44:29.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:29.712 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2023-07-14T17:44:29.729 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-07-14T17:44:29.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:29.730 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-07-14T17:44:29.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:29.804 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link delete ceph-brx 2023-07-14T17:44:29.900 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-07-14T17:44:29.901 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid b462a556-226c-11ee-9b33-001a4aab830c -- ceph fs ls 2023-07-14T17:44:30.458 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T17:44:31.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:31 smithi178 ceph-mon[129368]: pgmap v311: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-07-14T17:44:31.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:31 smithi088 ceph-mon[127465]: pgmap v311: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-07-14T17:44:31.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:31 smithi107 ceph-mon[128999]: pgmap v311: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-07-14T17:44:31.755 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.773+0000 7fb547209700 1 -- 172.21.15.88:0/4216915057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5400fffb0 msgr2=0x7fb540100380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:31.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.773+0000 7fb547209700 1 --2- 172.21.15.88:0/4216915057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5400fffb0 0x7fb540100380 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb5340099c0 tx=0x7fb534009cd0 comp rx=0 tx=0).stop 2023-07-14T17:44:31.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.773+0000 7fb547209700 1 -- 172.21.15.88:0/4216915057 shutdown_connections 2023-07-14T17:44:31.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.773+0000 7fb547209700 1 --2- 172.21.15.88:0/4216915057 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb540105000 0x7fb54010cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:31.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.773+0000 7fb547209700 1 --2- 172.21.15.88:0/4216915057 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5400fffb0 0x7fb540100380 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:31.756 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.773+0000 7fb547209700 1 --2- 172.21.15.88:0/4216915057 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb540100950 0x7fb5401048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:31.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.773+0000 7fb547209700 1 -- 172.21.15.88:0/4216915057 >> 172.21.15.88:0/4216915057 conn(0x7fb540077cd0 msgr2=0x7fb5400780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:31.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.773+0000 7fb547209700 1 -- 172.21.15.88:0/4216915057 shutdown_connections 2023-07-14T17:44:31.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.773+0000 7fb547209700 1 -- 172.21.15.88:0/4216915057 wait complete. 2023-07-14T17:44:31.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.774+0000 7fb547209700 1 Processor -- start 2023-07-14T17:44:31.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.774+0000 7fb547209700 1 -- start start 2023-07-14T17:44:31.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.774+0000 7fb547209700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5400fffb0 0x7fb54019b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:31.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.774+0000 7fb547209700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb540100950 0x7fb54019bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:31.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.774+0000 7fb547209700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb540105000 0x7fb540195a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:31.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.774+0000 7fb547209700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5401960b0 con 0x7fb5400fffb0 2023-07-14T17:44:31.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.774+0000 7fb547209700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb540196220 con 0x7fb540105000 2023-07-14T17:44:31.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.774+0000 7fb547209700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5401963e0 con 0x7fb540100950 2023-07-14T17:44:31.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.774+0000 7fb544fa5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5400fffb0 0x7fb54019b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:31.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.774+0000 7fb544fa5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5400fffb0 0x7fb54019b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43384/0 (socket says 172.21.15.88:43384) 2023-07-14T17:44:31.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.774+0000 7fb544fa5700 1 -- 172.21.15.88:0/2247891270 learned_addr learned my addr 172.21.15.88:0/2247891270 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:31.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.774+0000 7fb5457a6700 1 --2- 172.21.15.88:0/2247891270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb540105000 0x7fb540195a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:31.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.774+0000 7fb544fa5700 1 -- 172.21.15.88:0/2247891270 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb540100950 msgr2=0x7fb54019bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:31.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.775+0000 7fb53ffff700 1 --2- 172.21.15.88:0/2247891270 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb540100950 0x7fb54019bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:31.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.775+0000 7fb544fa5700 1 --2- 172.21.15.88:0/2247891270 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb540100950 0x7fb54019bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:31.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.775+0000 7fb544fa5700 1 -- 172.21.15.88:0/2247891270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb540105000 msgr2=0x7fb540195a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:31.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.775+0000 7fb544fa5700 1 --2- 172.21.15.88:0/2247891270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb540105000 0x7fb540195a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:31.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.775+0000 7fb544fa5700 1 -- 172.21.15.88:0/2247891270 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5340095f0 con 0x7fb5400fffb0 2023-07-14T17:44:31.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.775+0000 7fb5457a6700 1 --2- 172.21.15.88:0/2247891270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb540105000 0x7fb540195a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:31.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.775+0000 7fb544fa5700 1 --2- 172.21.15.88:0/2247891270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5400fffb0 0x7fb54019b9c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb534009f50 tx=0x7fb5340047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:31.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.775+0000 7fb53dffb700 1 -- 172.21.15.88:0/2247891270 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53401d070 con 0x7fb5400fffb0 2023-07-14T17:44:31.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.775+0000 7fb547209700 1 -- 172.21.15.88:0/2247891270 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb540196660 con 0x7fb5400fffb0 2023-07-14T17:44:31.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.775+0000 7fb53dffb700 1 -- 172.21.15.88:0/2247891270 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb534004040 con 0x7fb5400fffb0 2023-07-14T17:44:31.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.775+0000 7fb53dffb700 1 -- 172.21.15.88:0/2247891270 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb534022c80 con 0x7fb5400fffb0 2023-07-14T17:44:31.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.775+0000 7fb547209700 1 -- 172.21.15.88:0/2247891270 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5401a29b0 con 0x7fb5400fffb0 2023-07-14T17:44:31.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.776+0000 7fb547209700 1 -- 172.21.15.88:0/2247891270 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb54004e340 con 0x7fb5400fffb0 2023-07-14T17:44:31.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.777+0000 7fb53dffb700 1 -- 172.21.15.88:0/2247891270 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb534022470 con 0x7fb5400fffb0 2023-07-14T17:44:31.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.778+0000 7fb53dffb700 1 --2- 172.21.15.88:0/2247891270 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb5300a1150 0x7fb5300a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:31.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.778+0000 7fb53ffff700 1 --2- 172.21.15.88:0/2247891270 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb5300a1150 0x7fb5300a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:31.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.778+0000 7fb53dffb700 1 -- 172.21.15.88:0/2247891270 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb5340ce570 con 0x7fb5400fffb0 2023-07-14T17:44:31.763 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.778+0000 7fb53ffff700 1 --2- 172.21.15.88:0/2247891270 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb5300a1150 0x7fb5300a3600 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb5400fe1f0 tx=0x7fb52c005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:31.764 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:31.781+0000 7fb53dffb700 1 -- 172.21.15.88:0/2247891270 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb5340979e0 con 0x7fb5400fffb0 2023-07-14T17:44:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:32.009+0000 7fb547209700 1 -- 172.21.15.88:0/2247891270 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb5401a2c10 con 0x7fb5400fffb0 2023-07-14T17:44:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:32.009+0000 7fb53dffb700 1 -- 172.21.15.88:0/2247891270 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) v1 ==== 53+0+88 (secure 0 0 0) 0x7fb53409b000 con 0x7fb5400fffb0 2023-07-14T17:44:31.992 INFO:teuthology.orchestra.run.smithi088.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-07-14T17:44:31.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:32.011+0000 7fb547209700 1 -- 172.21.15.88:0/2247891270 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb5300a1150 msgr2=0x7fb5300a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:31.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:32.011+0000 7fb547209700 1 --2- 172.21.15.88:0/2247891270 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb5300a1150 0x7fb5300a3600 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb5400fe1f0 tx=0x7fb52c005d50 comp rx=0 tx=0).stop 2023-07-14T17:44:31.994 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:32.011+0000 7fb547209700 1 -- 172.21.15.88:0/2247891270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5400fffb0 msgr2=0x7fb54019b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:32.011+0000 7fb547209700 1 --2- 172.21.15.88:0/2247891270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5400fffb0 0x7fb54019b9c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb534009f50 tx=0x7fb5340047c0 comp rx=0 tx=0).stop 2023-07-14T17:44:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:32.011+0000 7fb547209700 1 -- 172.21.15.88:0/2247891270 shutdown_connections 2023-07-14T17:44:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:32.011+0000 7fb547209700 1 --2- 172.21.15.88:0/2247891270 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb540105000 0x7fb540195a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:32.011+0000 7fb547209700 1 --2- 172.21.15.88:0/2247891270 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb5300a1150 0x7fb5300a3600 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:32.011+0000 7fb547209700 1 --2- 172.21.15.88:0/2247891270 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb5400fffb0 0x7fb54019b9c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:32.011+0000 7fb547209700 1 --2- 172.21.15.88:0/2247891270 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb540100950 0x7fb54019bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:32.011+0000 7fb547209700 1 -- 172.21.15.88:0/2247891270 >> 172.21.15.88:0/2247891270 conn(0x7fb540077cd0 msgr2=0x7fb5400fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:32.011+0000 7fb547209700 1 -- 172.21.15.88:0/2247891270 shutdown_connections 2023-07-14T17:44:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:32.011+0000 7fb547209700 1 -- 172.21.15.88:0/2247891270 wait complete. 2023-07-14T17:44:32.353 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:32 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2247891270' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T17:44:32.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:32 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2247891270' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T17:44:32.414 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-07-14T17:44:32.415 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-14T17:44:32.415 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi088.front.sepia.ceph.com 2023-07-14T17:44:32.415 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-07-14T17:44:32.415 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-14T17:44:32.416 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-07-14T17:44:32.416 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-07-14T17:44:32.416 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-14T17:44:32.416 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-07-14T17:44:32.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:32.417 DEBUG:teuthology.orchestra.run.smithi088:> ip addr 2023-07-14T17:44:32.435 INFO:teuthology.orchestra.run.smithi088.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-14T17:44:32.436 INFO:teuthology.orchestra.run.smithi088.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-14T17:44:32.436 INFO:teuthology.orchestra.run.smithi088.stdout: inet 127.0.0.1/8 scope host lo 2023-07-14T17:44:32.436 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2023-07-14T17:44:32.436 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 ::1/128 scope host 2023-07-14T17:44:32.436 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2023-07-14T17:44:32.437 INFO:teuthology.orchestra.run.smithi088.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T17:44:32.437 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f0 brd ff:ff:ff:ff:ff:ff 2023-07-14T17:44:32.437 INFO:teuthology.orchestra.run.smithi088.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T17:44:32.437 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:d8:a8:f1 brd ff:ff:ff:ff:ff:ff 2023-07-14T17:44:32.437 INFO:teuthology.orchestra.run.smithi088.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T17:44:32.437 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2c brd ff:ff:ff:ff:ff:ff 2023-07-14T17:44:32.438 INFO:teuthology.orchestra.run.smithi088.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-14T17:44:32.438 INFO:teuthology.orchestra.run.smithi088.stdout: link/ether 0c:c4:7a:8f:d4:2d brd ff:ff:ff:ff:ff:ff 2023-07-14T17:44:32.438 INFO:teuthology.orchestra.run.smithi088.stdout: inet 172.21.15.88/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-07-14T17:44:32.438 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft 41420sec preferred_lft 41420sec 2023-07-14T17:44:32.438 INFO:teuthology.orchestra.run.smithi088.stdout: inet6 fe80::4893:55f2:c841:20/64 scope link noprefixroute 2023-07-14T17:44:32.439 INFO:teuthology.orchestra.run.smithi088.stdout: valid_lft forever preferred_lft forever 2023-07-14T17:44:32.439 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-07-14T17:44:32.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:32.440 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-07-14T17:44:32.440 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-07-14T17:44:32.440 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link add name ceph-brx type bridge 2023-07-14T17:44:32.440 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip addr flush dev ceph-brx 2023-07-14T17:44:32.441 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set ceph-brx up 2023-07-14T17:44:32.441 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-07-14T17:44:32.441 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-07-14T17:44:32.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:32 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2247891270' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T17:44:32.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:32.555 DEBUG:teuthology.orchestra.run.smithi088:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-14T17:44:32.581 INFO:teuthology.orchestra.run.smithi088.stdout:1 2023-07-14T17:44:32.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:32.583 DEBUG:teuthology.orchestra.run.smithi088:> route 2023-07-14T17:44:32.642 INFO:teuthology.orchestra.run.smithi088.stdout:Kernel IP routing table 2023-07-14T17:44:32.643 INFO:teuthology.orchestra.run.smithi088.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-14T17:44:32.643 INFO:teuthology.orchestra.run.smithi088.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-07-14T17:44:32.643 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-07-14T17:44:32.643 INFO:teuthology.orchestra.run.smithi088.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-14T17:44:32.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:32.644 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-07-14T17:44:32.644 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-07-14T17:44:32.644 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-07-14T17:44:32.644 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-07-14T17:44:32.645 DEBUG:teuthology.orchestra.run.smithi088:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-07-14T17:44:32.645 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-07-14T17:44:32.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:32.747 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2023-07-14T17:44:32.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:32.765 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list-id 2023-07-14T17:44:32.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:32.821 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-07-14T17:44:32.822 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-07-14T17:44:32.822 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-14T17:44:32.822 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-07-14T17:44:32.822 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-07-14T17:44:32.915 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-07-14T17:44:32.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:32.915 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-07-14T17:44:32.916 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-07-14T17:44:32.916 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-07-14T17:44:32.916 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-07-14T17:44:32.916 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-07-14T17:44:32.917 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-07-14T17:44:32.917 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-07-14T17:44:32.917 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-07-14T17:44:33.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:33.129 DEBUG:teuthology.orchestra.run.smithi088:> (cd / && exec bash -c ' 2023-07-14T17:44:33.129 DEBUG:teuthology.orchestra.run.smithi088:> set -e 2023-07-14T17:44:33.129 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set brx.0 up 2023-07-14T17:44:33.130 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link set dev brx.0 master ceph-brx 2023-07-14T17:44:33.130 DEBUG:teuthology.orchestra.run.smithi088:> ') 2023-07-14T17:44:33.243 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2023-07-14T17:44:33.244 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T17:44:33.244 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-07-14T17:44:33.250 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:33 smithi088 ceph-mon[127465]: pgmap v312: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2023-07-14T17:44:33.266 INFO:teuthology.orchestra.run.smithi088.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-07-14T17:44:33.267 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T17:44:33.267 DEBUG:teuthology.orchestra.run.smithi088:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-07-14T17:44:33.323 DEBUG:teuthology.orchestra.run.smithi088:> sudo modprobe fuse 2023-07-14T17:44:33.391 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-14T17:44:33.391 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-14T17:44:33.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:33 smithi178 ceph-mon[129368]: pgmap v312: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2023-07-14T17:44:33.457 INFO:teuthology.orchestra.run.smithi088.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-14T17:44:33.458 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-14T17:44:33.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:33.459 DEBUG:teuthology.orchestra.run.smithi088:> ls /sys/fs/fuse/connections 2023-07-14T17:44:33.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:33 smithi107 ceph-mon[128999]: pgmap v312: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2023-07-14T17:44:33.515 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-07-14T17:44:33.516 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/_nogroup/sv_0/9513faeb-71d8-4552-bbbd-ba63e6a250d0 --client_fs=cephfs) 2023-07-14T17:44:33.558 DEBUG:teuthology.orchestra.run.smithi088:> sudo modprobe fuse 2023-07-14T17:44:33.585 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-14T17:44:33.585 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-14T17:44:33.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi088.stderr:2023-07-14T17:44:33.664+0000 7fd4a8c115c0 -1 init, newargv = 0x55f35021fb90 newargc=15 2023-07-14T17:44:33.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi088.stderr:ceph-fuse[178582]: starting ceph client 2023-07-14T17:44:33.652 INFO:teuthology.orchestra.run.smithi088.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-14T17:44:33.653 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-14T17:44:33.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:33.654 DEBUG:teuthology.orchestra.run.smithi088:> ls /sys/fs/fuse/connections 2023-07-14T17:44:33.668 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi088.stderr:ceph-fuse[178582]: starting fuse 2023-07-14T17:44:33.684 INFO:teuthology.orchestra.run.smithi088.stdout:124 2023-07-14T17:44:33.684 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [124] 2023-07-14T17:44:33.685 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-07-14T17:44:33.685 DEBUG:teuthology.orchestra.run.smithi088:> import glob 2023-07-14T17:44:33.685 DEBUG:teuthology.orchestra.run.smithi088:> import re 2023-07-14T17:44:33.685 DEBUG:teuthology.orchestra.run.smithi088:> import os 2023-07-14T17:44:33.686 DEBUG:teuthology.orchestra.run.smithi088:> import subprocess 2023-07-14T17:44:33.686 DEBUG:teuthology.orchestra.run.smithi088:> 2023-07-14T17:44:33.686 DEBUG:teuthology.orchestra.run.smithi088:> def _find_admin_socket(client_name): 2023-07-14T17:44:33.686 DEBUG:teuthology.orchestra.run.smithi088:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-07-14T17:44:33.686 DEBUG:teuthology.orchestra.run.smithi088:> files = glob.glob(asok_path) 2023-07-14T17:44:33.687 DEBUG:teuthology.orchestra.run.smithi088:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-07-14T17:44:33.687 DEBUG:teuthology.orchestra.run.smithi088:> 2023-07-14T17:44:33.687 DEBUG:teuthology.orchestra.run.smithi088:> # Given a non-glob path, it better be there 2023-07-14T17:44:33.687 DEBUG:teuthology.orchestra.run.smithi088:> if "*" not in asok_path: 2023-07-14T17:44:33.688 DEBUG:teuthology.orchestra.run.smithi088:> assert(len(files) == 1) 2023-07-14T17:44:33.688 DEBUG:teuthology.orchestra.run.smithi088:> return files[0] 2023-07-14T17:44:33.688 DEBUG:teuthology.orchestra.run.smithi088:> 2023-07-14T17:44:33.688 DEBUG:teuthology.orchestra.run.smithi088:> for f in files: 2023-07-14T17:44:33.688 DEBUG:teuthology.orchestra.run.smithi088:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-07-14T17:44:33.689 DEBUG:teuthology.orchestra.run.smithi088:> if os.path.exists("/proc/{0}".format(pid)): 2023-07-14T17:44:33.689 DEBUG:teuthology.orchestra.run.smithi088:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-07-14T17:44:33.689 DEBUG:teuthology.orchestra.run.smithi088:> contents = proc_f.read() 2023-07-14T17:44:33.689 DEBUG:teuthology.orchestra.run.smithi088:> if mountpoint in contents: 2023-07-14T17:44:33.690 DEBUG:teuthology.orchestra.run.smithi088:> return f 2023-07-14T17:44:33.690 DEBUG:teuthology.orchestra.run.smithi088:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-07-14T17:44:33.690 DEBUG:teuthology.orchestra.run.smithi088:> 2023-07-14T17:44:33.690 DEBUG:teuthology.orchestra.run.smithi088:> print(_find_admin_socket("client.0")) 2023-07-14T17:44:33.691 DEBUG:teuthology.orchestra.run.smithi088:> ' 2023-07-14T17:44:33.808 INFO:teuthology.orchestra.run.smithi088.stdout:/var/run/ceph/ceph-client.0.178582.asok 2023-07-14T17:44:33.990 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.178582.asok 2023-07-14T17:44:33.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:33.991 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.178582.asok status 2023-07-14T17:44:34.148 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-07-14T17:44:34.148 INFO:teuthology.orchestra.run.smithi088.stdout: "metadata": { 2023-07-14T17:44:34.148 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_sha1": "81c9387d2e7576da065747ec9c16d1c7d69016ec", 2023-07-14T17:44:34.148 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T17:44:34.148 INFO:teuthology.orchestra.run.smithi088.stdout: "entity_id": "0", 2023-07-14T17:44:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: "hostname": "smithi088", 2023-07-14T17:44:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-07-14T17:44:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: "pid": "178582", 2023-07-14T17:44:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: "root": "/volumes/_nogroup/sv_0/9513faeb-71d8-4552-bbbd-ba63e6a250d0" 2023-07-14T17:44:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-07-14T17:44:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: "dentry_count": 0, 2023-07-14T17:44:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: "dentry_pinned_count": 0, 2023-07-14T17:44:34.149 INFO:teuthology.orchestra.run.smithi088.stdout: "id": 15531, 2023-07-14T17:44:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: "inst": { 2023-07-14T17:44:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: "name": { 2023-07-14T17:44:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "client", 2023-07-14T17:44:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: "num": 15531 2023-07-14T17:44:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-07-14T17:44:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": { 2023-07-14T17:44:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2023-07-14T17:44:34.150 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "192.168.0.1:0", 2023-07-14T17:44:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1784556328 2023-07-14T17:44:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: } 2023-07-14T17:44:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-07-14T17:44:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": { 2023-07-14T17:44:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: "type": "v1", 2023-07-14T17:44:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: "addr": "192.168.0.1:0", 2023-07-14T17:44:34.151 INFO:teuthology.orchestra.run.smithi088.stdout: "nonce": 1784556328 2023-07-14T17:44:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: }, 2023-07-14T17:44:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "inst_str": "client.15531 192.168.0.1:0/1784556328", 2023-07-14T17:44:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "addr_str": "192.168.0.1:0/1784556328", 2023-07-14T17:44:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "inode_count": 5, 2023-07-14T17:44:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "mds_epoch": 27, 2023-07-14T17:44:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_epoch": 88, 2023-07-14T17:44:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "osd_epoch_barrier": 0, 2023-07-14T17:44:34.152 INFO:teuthology.orchestra.run.smithi088.stdout: "blocklisted": false, 2023-07-14T17:44:34.153 INFO:teuthology.orchestra.run.smithi088.stdout: "fs_name": "cephfs" 2023-07-14T17:44:34.153 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-07-14T17:44:34.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:34.158 DEBUG:teuthology.orchestra.run.smithi088:> stat --file-system '--printf=%T 2023-07-14T17:44:34.158 DEBUG:teuthology.orchestra.run.smithi088:> ' -- /home/ubuntu/cephtest/mnt.0 2023-07-14T17:44:34.175 INFO:teuthology.orchestra.run.smithi088.stdout:fuseblk 2023-07-14T17:44:34.176 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-07-14T17:44:34.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:34.176 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-07-14T17:44:34.245 INFO:teuthology.run_tasks:Running task exec... 2023-07-14T17:44:34.255 INFO:teuthology.task.exec:Executing custom commands... 2023-07-14T17:44:34.256 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi088.front.sepia.ceph.com 2023-07-14T17:44:34.256 DEBUG:teuthology.orchestra.run.smithi088:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2023-07-14T17:44:34.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.423+0000 7fd46b169700 1 -- 172.21.15.88:0/1772502721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4640feb30 msgr2=0x7fd4640fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:34.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.423+0000 7fd46b169700 1 --2- 172.21.15.88:0/1772502721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4640feb30 0x7fd4640fef40 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd458009990 tx=0x7fd458009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:34.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.424+0000 7fd46b169700 1 -- 172.21.15.88:0/1772502721 shutdown_connections 2023-07-14T17:44:34.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.424+0000 7fd46b169700 1 --2- 172.21.15.88:0/1772502721 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd464100770 0x7fd464107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:34.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.424+0000 7fd46b169700 1 --2- 172.21.15.88:0/1772502721 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4640feb30 0x7fd4640fef40 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:34.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.424+0000 7fd46b169700 1 --2- 172.21.15.88:0/1772502721 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4640ffdc0 0x7fd464100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:34.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.424+0000 7fd46b169700 1 -- 172.21.15.88:0/1772502721 >> 172.21.15.88:0/1772502721 conn(0x7fd4640fa130 msgr2=0x7fd4640fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:34.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.424+0000 7fd46b169700 1 -- 172.21.15.88:0/1772502721 shutdown_connections 2023-07-14T17:44:34.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.424+0000 7fd46b169700 1 -- 172.21.15.88:0/1772502721 wait complete. 2023-07-14T17:44:34.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.424+0000 7fd46b169700 1 Processor -- start 2023-07-14T17:44:34.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.424+0000 7fd46b169700 1 -- start start 2023-07-14T17:44:34.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.424+0000 7fd46b169700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4640feb30 0x7fd4641985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:34.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.424+0000 7fd46b169700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4640ffdc0 0x7fd46419aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:34.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.424+0000 7fd46b169700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd464100770 0x7fd46419d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:34.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.424+0000 7fd46b169700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46419d610 con 0x7fd4640feb30 2023-07-14T17:44:34.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.424+0000 7fd46b169700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46419d750 con 0x7fd464100770 2023-07-14T17:44:34.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.424+0000 7fd46b169700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46419d910 con 0x7fd4640ffdc0 2023-07-14T17:44:34.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.425+0000 7fd468f05700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4640feb30 0x7fd4641985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:34.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.425+0000 7fd468f05700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4640feb30 0x7fd4641985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43418/0 (socket says 172.21.15.88:43418) 2023-07-14T17:44:34.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.425+0000 7fd468f05700 1 -- 172.21.15.88:0/2703033925 learned_addr learned my addr 172.21.15.88:0/2703033925 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:34.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.425+0000 7fd463fff700 1 --2- 172.21.15.88:0/2703033925 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4640ffdc0 0x7fd46419aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:34.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.425+0000 7fd468f05700 1 -- 172.21.15.88:0/2703033925 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4640ffdc0 msgr2=0x7fd46419aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:34.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.425+0000 7fd468f05700 1 --2- 172.21.15.88:0/2703033925 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4640ffdc0 0x7fd46419aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:34.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.425+0000 7fd468f05700 1 -- 172.21.15.88:0/2703033925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd464100770 msgr2=0x7fd46419d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:34.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.425+0000 7fd468f05700 1 --2- 172.21.15.88:0/2703033925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd464100770 0x7fd46419d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:34.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.425+0000 7fd468f05700 1 -- 172.21.15.88:0/2703033925 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4580095f0 con 0x7fd4640feb30 2023-07-14T17:44:34.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.425+0000 7fd468f05700 1 --2- 172.21.15.88:0/2703033925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4640feb30 0x7fd4641985a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd4580059d0 tx=0x7fd4580040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:34.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.425+0000 7fd461ffb700 1 -- 172.21.15.88:0/2703033925 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45801d070 con 0x7fd4640feb30 2023-07-14T17:44:34.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.425+0000 7fd461ffb700 1 -- 172.21.15.88:0/2703033925 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd4580041d0 con 0x7fd4640feb30 2023-07-14T17:44:34.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.425+0000 7fd46b169700 1 -- 172.21.15.88:0/2703033925 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd46419db90 con 0x7fd4640feb30 2023-07-14T17:44:34.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.425+0000 7fd461ffb700 1 -- 172.21.15.88:0/2703033925 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd458021410 con 0x7fd4640feb30 2023-07-14T17:44:34.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.425+0000 7fd46b169700 1 -- 172.21.15.88:0/2703033925 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd46419e040 con 0x7fd4640feb30 2023-07-14T17:44:34.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.426+0000 7fd46b169700 1 -- 172.21.15.88:0/2703033925 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd46404e420 con 0x7fd4640feb30 2023-07-14T17:44:34.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.427+0000 7fd461ffb700 1 -- 172.21.15.88:0/2703033925 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd45800f460 con 0x7fd4640feb30 2023-07-14T17:44:34.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.428+0000 7fd461ffb700 1 --2- 172.21.15.88:0/2703033925 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd4540a1150 0x7fd4540a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:34.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.428+0000 7fd463fff700 1 --2- 172.21.15.88:0/2703033925 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd4540a1150 0x7fd4540a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:34.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.428+0000 7fd461ffb700 1 -- 172.21.15.88:0/2703033925 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd4580ce3b0 con 0x7fd4640feb30 2023-07-14T17:44:34.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.428+0000 7fd463fff700 1 --2- 172.21.15.88:0/2703033925 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd4540a1150 0x7fd4540a3600 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd464067f00 tx=0x7fd450009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:34.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.432+0000 7fd461ffb700 1 -- 172.21.15.88:0/2703033925 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd458097880 con 0x7fd4640feb30 2023-07-14T17:44:34.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:34.669+0000 7fd46b169700 1 -- 172.21.15.88:0/2703033925 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7fd464105b40 con 0x7fd4640feb30 2023-07-14T17:44:35.023 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.041+0000 7fd461ffb700 1 -- 172.21.15.88:0/2703033925 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v28) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd45809aea0 con 0x7fd4640feb30 2023-07-14T17:44:35.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.043+0000 7fd46b169700 1 -- 172.21.15.88:0/2703033925 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd4540a1150 msgr2=0x7fd4540a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:35.025 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.043+0000 7fd46b169700 1 --2- 172.21.15.88:0/2703033925 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd4540a1150 0x7fd4540a3600 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd464067f00 tx=0x7fd450009260 comp rx=0 tx=0).stop 2023-07-14T17:44:35.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.043+0000 7fd46b169700 1 -- 172.21.15.88:0/2703033925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4640feb30 msgr2=0x7fd4641985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:35.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.043+0000 7fd46b169700 1 --2- 172.21.15.88:0/2703033925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4640feb30 0x7fd4641985a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd4580059d0 tx=0x7fd4580040b0 comp rx=0 tx=0).stop 2023-07-14T17:44:35.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.044+0000 7fd46b169700 1 -- 172.21.15.88:0/2703033925 shutdown_connections 2023-07-14T17:44:35.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.044+0000 7fd46b169700 1 --2- 172.21.15.88:0/2703033925 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd464100770 0x7fd46419d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:35.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.044+0000 7fd46b169700 1 --2- 172.21.15.88:0/2703033925 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd4540a1150 0x7fd4540a3600 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:35.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.044+0000 7fd46b169700 1 --2- 172.21.15.88:0/2703033925 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4640feb30 0x7fd4641985a0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:35.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.044+0000 7fd46b169700 1 --2- 172.21.15.88:0/2703033925 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4640ffdc0 0x7fd46419aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:35.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.044+0000 7fd46b169700 1 -- 172.21.15.88:0/2703033925 >> 172.21.15.88:0/2703033925 conn(0x7fd4640fa130 msgr2=0x7fd4640fc450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:35.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.044+0000 7fd46b169700 1 -- 172.21.15.88:0/2703033925 shutdown_connections 2023-07-14T17:44:35.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.044+0000 7fd46b169700 1 -- 172.21.15.88:0/2703033925 wait complete. 2023-07-14T17:44:35.038 DEBUG:teuthology.orchestra.run.smithi088:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup random 0.01' 2023-07-14T17:44:35.162 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.181+0000 7f3a758b4700 1 -- 172.21.15.88:0/132359221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a700fdcc0 msgr2=0x7f3a700fe110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.181+0000 7f3a758b4700 1 --2- 172.21.15.88:0/132359221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a700fdcc0 0x7f3a700fe110 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009990 tx=0x7f3a60009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.181+0000 7f3a758b4700 1 -- 172.21.15.88:0/132359221 shutdown_connections 2023-07-14T17:44:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.181+0000 7f3a758b4700 1 --2- 172.21.15.88:0/132359221 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a700fe650 0x7f3a70108230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.181+0000 7f3a758b4700 1 --2- 172.21.15.88:0/132359221 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a70101250 0x7f3a70101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.181+0000 7f3a758b4700 1 --2- 172.21.15.88:0/132359221 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a700fdcc0 0x7f3a700fe110 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.181+0000 7f3a758b4700 1 -- 172.21.15.88:0/132359221 >> 172.21.15.88:0/132359221 conn(0x7f3a700f98f0 msgr2=0x7f3a700fbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:35.163 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.181+0000 7f3a758b4700 1 -- 172.21.15.88:0/132359221 shutdown_connections 2023-07-14T17:44:35.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.181+0000 7f3a758b4700 1 -- 172.21.15.88:0/132359221 wait complete. 2023-07-14T17:44:35.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.181+0000 7f3a758b4700 1 Processor -- start 2023-07-14T17:44:35.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.181+0000 7f3a758b4700 1 -- start start 2023-07-14T17:44:35.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.181+0000 7f3a758b4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a700fdcc0 0x7f3a70195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:35.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.181+0000 7f3a758b4700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a700fe650 0x7f3a70195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:35.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.181+0000 7f3a758b4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a70101250 0x7f3a70196d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:35.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a758b4700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a701973a0 con 0x7f3a70101250 2023-07-14T17:44:35.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a758b4700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a70197510 con 0x7f3a700fdcc0 2023-07-14T17:44:35.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a758b4700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a701976d0 con 0x7f3a700fe650 2023-07-14T17:44:35.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a6f7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a70101250 0x7f3a70196d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:35.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a6f7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a70101250 0x7f3a70196d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43434/0 (socket says 172.21.15.88:43434) 2023-07-14T17:44:35.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a6f7fe700 1 -- 172.21.15.88:0/2830336126 learned_addr learned my addr 172.21.15.88:0/2830336126 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:35.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a6effd700 1 --2- 172.21.15.88:0/2830336126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a700fdcc0 0x7f3a70195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:35.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a6e7fc700 1 --2- 172.21.15.88:0/2830336126 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a700fe650 0x7f3a70195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:35.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a6f7fe700 1 -- 172.21.15.88:0/2830336126 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a700fe650 msgr2=0x7f3a70195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:35.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a6f7fe700 1 --2- 172.21.15.88:0/2830336126 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f3a700fe650 0x7f3a70195ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:35.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a6f7fe700 1 -- 172.21.15.88:0/2830336126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a700fdcc0 msgr2=0x7f3a70195980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:35.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a6f7fe700 1 --2- 172.21.15.88:0/2830336126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f3a700fdcc0 0x7f3a70195980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:35.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a6f7fe700 1 -- 172.21.15.88:0/2830336126 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a600095f0 con 0x7f3a70101250 2023-07-14T17:44:35.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a6f7fe700 1 --2- 172.21.15.88:0/2830336126 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3a70101250 0x7f3a70196d40 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800ef30 tx=0x7f3a6800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:35.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a748b2700 1 -- 172.21.15.88:0/2830336126 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a68018070 con 0x7f3a70101250 2023-07-14T17:44:35.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a748b2700 1 -- 172.21.15.88:0/2830336126 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3a6800f040 con 0x7f3a70101250 2023-07-14T17:44:35.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a758b4700 1 -- 172.21.15.88:0/2830336126 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a70197950 con 0x7f3a70101250 2023-07-14T17:44:35.168 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a748b2700 1 -- 172.21.15.88:0/2830336126 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6800cc70 con 0x7f3a70101250 2023-07-14T17:44:35.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.182+0000 7f3a758b4700 1 -- 172.21.15.88:0/2830336126 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a7019e170 con 0x7f3a70101250 2023-07-14T17:44:35.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.183+0000 7f3a758b4700 1 -- 172.21.15.88:0/2830336126 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a700fd580 con 0x7f3a70101250 2023-07-14T17:44:35.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.184+0000 7f3a748b2700 1 -- 172.21.15.88:0/2830336126 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3a68003a00 con 0x7f3a70101250 2023-07-14T17:44:35.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.185+0000 7f3a748b2700 1 --2- 172.21.15.88:0/2830336126 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3a580a1150 0x7f3a580a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:35.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.185+0000 7f3a6effd700 1 --2- 172.21.15.88:0/2830336126 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3a580a1150 0x7f3a580a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:35.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.185+0000 7f3a748b2700 1 -- 172.21.15.88:0/2830336126 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3a680ccd90 con 0x7f3a70101250 2023-07-14T17:44:35.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.185+0000 7f3a6effd700 1 --2- 172.21.15.88:0/2830336126 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f3a580a1150 0x7f3a580a3600 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3a70068660 tx=0x7f3a6400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:35.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.188+0000 7f3a748b2700 1 -- 172.21.15.88:0/2830336126 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3a68096200 con 0x7f3a70101250 2023-07-14T17:44:35.390 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.408+0000 7f3a758b4700 1 -- 172.21.15.88:0/2830336126 --> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}) v1 -- 0x7f3a70060b30 con 0x7f3a580a1150 2023-07-14T17:44:35.395 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:35 smithi088 ceph-mon[127465]: pgmap v313: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2023-07-14T17:44:35.396 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:35 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2703033925' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T17:44:35.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.426+0000 7f3a748b2700 1 -- 172.21.15.88:0/2830336126 <== mgr.14152 v2:172.21.15.88:6800/4262759610 1 ==== mgr_command_reply(tid 0: -1 Operation not permitted for group '_nogroup' as it is an internal group.) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3a70060b30 con 0x7f3a580a1150 2023-07-14T17:44:35.409 INFO:teuthology.orchestra.run.smithi088.stderr:Error EPERM: Operation not permitted for group '_nogroup' as it is an internal group. 2023-07-14T17:44:35.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:44:35.416 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup random 0.01'" 2023-07-14T17:44:35.529 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ff352723e68348babba8dbca4ff1b611 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup random 0.01'" 2023-07-14T17:44:35.531 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-07-14T17:44:35.540 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-07-14T17:44:35.540 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:44:35.541 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T17:44:35.561 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:35 smithi107 ceph-mon[128999]: pgmap v313: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2023-07-14T17:44:35.561 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:35 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2703033925' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T17:44:35.561 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:35 smithi178 ceph-mon[129368]: pgmap v313: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2023-07-14T17:44:35.562 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:35 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2703033925' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T17:44:35.578 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:44:35.578 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T17:44:35.635 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:44:35.690 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:44:35 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:44:35.413+0000 7f4490462700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T17:44:35.691 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:44:35 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:44:35.413+0000 7f4490462700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T17:44:35.691 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:44:35 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:44:35.413+0000 7f4490462700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T17:44:35.691 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:44:35 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:44:35.413+0000 7f4490462700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T17:44:35.692 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:44:35 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:44:35.413+0000 7f4490462700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T17:44:35.692 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:44:35 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:44:35.426+0000 7f4490462700 -1 mgr.server reply reply (1) Operation not permitted Operation not permitted for group '_nogroup' as it is an internal group. 2023-07-14T17:44:35.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.816+0000 7fde3bea4700 1 -- 172.21.15.88:0/660788241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde34100700 msgr2=0x7fde34102ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:35.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.816+0000 7fde3bea4700 1 --2- 172.21.15.88:0/660788241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde34100700 0x7fde34102ae0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fde30009990 tx=0x7fde30009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:35.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.816+0000 7fde3bea4700 1 -- 172.21.15.88:0/660788241 shutdown_connections 2023-07-14T17:44:35.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.816+0000 7fde3bea4700 1 --2- 172.21.15.88:0/660788241 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde34105940 0x7fde34107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:35.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.816+0000 7fde3bea4700 1 --2- 172.21.15.88:0/660788241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde34100700 0x7fde34102ae0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:35.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.816+0000 7fde3bea4700 1 --2- 172.21.15.88:0/660788241 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde34103020 0x7fde34105400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:35.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.816+0000 7fde3bea4700 1 -- 172.21.15.88:0/660788241 >> 172.21.15.88:0/660788241 conn(0x7fde340fa100 msgr2=0x7fde340fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:35.800 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.816+0000 7fde3bea4700 1 -- 172.21.15.88:0/660788241 shutdown_connections 2023-07-14T17:44:35.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.816+0000 7fde3bea4700 1 -- 172.21.15.88:0/660788241 wait complete. 2023-07-14T17:44:35.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde3bea4700 1 Processor -- start 2023-07-14T17:44:35.801 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde3bea4700 1 -- start start 2023-07-14T17:44:35.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde3bea4700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde34100700 0x7fde34100070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:35.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde3bea4700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde34103020 0x7fde340fe6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:35.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde3bea4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde34105940 0x7fde340febf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:35.806 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde3bea4700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde340ff390 con 0x7fde34105940 2023-07-14T17:44:35.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde3bea4700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde340ff500 con 0x7fde34100700 2023-07-14T17:44:35.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde3bea4700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3419f910 con 0x7fde34103020 2023-07-14T17:44:35.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde3a441700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde34105940 0x7fde340febf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:35.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde3a441700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde34105940 0x7fde340febf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43468/0 (socket says 172.21.15.88:43468) 2023-07-14T17:44:35.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde3a441700 1 -- 172.21.15.88:0/2886957447 learned_addr learned my addr 172.21.15.88:0/2886957447 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:35.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde3943f700 1 --2- 172.21.15.88:0/2886957447 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde34103020 0x7fde340fe6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:35.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde39c40700 1 --2- 172.21.15.88:0/2886957447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde34100700 0x7fde34100070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:35.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde3a441700 1 -- 172.21.15.88:0/2886957447 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde34103020 msgr2=0x7fde340fe6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:35.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde3a441700 1 --2- 172.21.15.88:0/2886957447 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde34103020 0x7fde340fe6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:35.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde3a441700 1 -- 172.21.15.88:0/2886957447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde34100700 msgr2=0x7fde34100070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:35.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde3a441700 1 --2- 172.21.15.88:0/2886957447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde34100700 0x7fde34100070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:35.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde3a441700 1 -- 172.21.15.88:0/2886957447 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde300095f0 con 0x7fde34105940 2023-07-14T17:44:35.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.817+0000 7fde39c40700 1 --2- 172.21.15.88:0/2886957447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde34100700 0x7fde34100070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:35.811 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.818+0000 7fde3a441700 1 --2- 172.21.15.88:0/2886957447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde34105940 0x7fde340febf0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fde2c007ee0 tx=0x7fde2c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:35.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.818+0000 7fde26ffd700 1 -- 172.21.15.88:0/2886957447 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2c017070 con 0x7fde34105940 2023-07-14T17:44:35.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.818+0000 7fde26ffd700 1 -- 172.21.15.88:0/2886957447 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fde2c00f040 con 0x7fde34105940 2023-07-14T17:44:35.812 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.818+0000 7fde3bea4700 1 -- 172.21.15.88:0/2886957447 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde3419fbf0 con 0x7fde34105940 2023-07-14T17:44:35.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.818+0000 7fde26ffd700 1 -- 172.21.15.88:0/2886957447 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2c005710 con 0x7fde34105940 2023-07-14T17:44:35.813 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.818+0000 7fde3bea4700 1 -- 172.21.15.88:0/2886957447 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde341a00b0 con 0x7fde34105940 2023-07-14T17:44:35.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.819+0000 7fde3bea4700 1 -- 172.21.15.88:0/2886957447 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde340fbc70 con 0x7fde34105940 2023-07-14T17:44:35.814 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.820+0000 7fde26ffd700 1 -- 172.21.15.88:0/2886957447 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fde2c005870 con 0x7fde34105940 2023-07-14T17:44:35.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.820+0000 7fde26ffd700 1 --2- 172.21.15.88:0/2886957447 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fde200a11a0 0x7fde200a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:35.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.820+0000 7fde26ffd700 1 -- 172.21.15.88:0/2886957447 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fde2c0cc430 con 0x7fde34105940 2023-07-14T17:44:35.815 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.820+0000 7fde39c40700 1 --2- 172.21.15.88:0/2886957447 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fde200a11a0 0x7fde200a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:35.816 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.821+0000 7fde39c40700 1 --2- 172.21.15.88:0/2886957447 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fde200a11a0 0x7fde200a3650 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fde3000b5c0 tx=0x7fde30000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:35.817 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:35.824+0000 7fde26ffd700 1 -- 172.21.15.88:0/2886957447 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fde2c095870 con 0x7fde34105940 2023-07-14T17:44:35.996 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.014+0000 7fde3bea4700 1 -- 172.21.15.88:0/2886957447 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde3402c8e0 con 0x7fde34105940 2023-07-14T17:44:35.997 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.016+0000 7fde26ffd700 1 -- 172.21.15.88:0/2886957447 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24512 (secure 0 0 0) 0x7fde2c098e90 con 0x7fde34105940 2023-07-14T17:44:35.998 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:35.998 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":88,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:43:44.887959+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:44:35.999 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.018+0000 7fde3bea4700 1 -- 172.21.15.88:0/2886957447 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fde200a11a0 msgr2=0x7fde200a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:36.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.018+0000 7fde3bea4700 1 --2- 172.21.15.88:0/2886957447 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fde200a11a0 0x7fde200a3650 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fde3000b5c0 tx=0x7fde30000bc0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.018+0000 7fde3bea4700 1 -- 172.21.15.88:0/2886957447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde34105940 msgr2=0x7fde340febf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:36.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.018+0000 7fde3bea4700 1 --2- 172.21.15.88:0/2886957447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde34105940 0x7fde340febf0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fde2c007ee0 tx=0x7fde2c00d3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.000 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.018+0000 7fde3bea4700 1 -- 172.21.15.88:0/2886957447 shutdown_connections 2023-07-14T17:44:36.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.018+0000 7fde3bea4700 1 --2- 172.21.15.88:0/2886957447 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fde200a11a0 0x7fde200a3650 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.018+0000 7fde3bea4700 1 --2- 172.21.15.88:0/2886957447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde34105940 0x7fde340febf0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.018+0000 7fde3bea4700 1 --2- 172.21.15.88:0/2886957447 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fde34100700 0x7fde34100070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.018+0000 7fde3bea4700 1 --2- 172.21.15.88:0/2886957447 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fde34103020 0x7fde340fe6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.001 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.018+0000 7fde3bea4700 1 -- 172.21.15.88:0/2886957447 >> 172.21.15.88:0/2886957447 conn(0x7fde340fa100 msgr2=0x7fde34101360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:36.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.018+0000 7fde3bea4700 1 -- 172.21.15.88:0/2886957447 shutdown_connections 2023-07-14T17:44:36.002 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.018+0000 7fde3bea4700 1 -- 172.21.15.88:0/2886957447 wait complete. 2023-07-14T17:44:36.012 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T17:38:50.126535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T17:42:40.725870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T17:42:41.512749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T17:42:44.764573+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T17:44:36.013 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T17:44:36.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.193+0000 7fc08c014700 1 -- 172.21.15.88:0/2820770590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0840ffd20 msgr2=0x7fc084100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:36.178 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.193+0000 7fc08c014700 1 --2- 172.21.15.88:0/2820770590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0840ffd20 0x7fc084100170 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc074009990 tx=0x7fc074009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.193+0000 7fc08c014700 1 -- 172.21.15.88:0/2820770590 shutdown_connections 2023-07-14T17:44:36.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.193+0000 7fc08c014700 1 --2- 172.21.15.88:0/2820770590 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0841006b0 0x7fc084106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.179 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.193+0000 7fc08c014700 1 --2- 172.21.15.88:0/2820770590 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0840feb30 0x7fc0840fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.193+0000 7fc08c014700 1 --2- 172.21.15.88:0/2820770590 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0840ffd20 0x7fc084100170 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.193+0000 7fc08c014700 1 -- 172.21.15.88:0/2820770590 >> 172.21.15.88:0/2820770590 conn(0x7fc0840fa190 msgr2=0x7fc0840fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:36.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.193+0000 7fc08c014700 1 -- 172.21.15.88:0/2820770590 shutdown_connections 2023-07-14T17:44:36.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.193+0000 7fc08c014700 1 -- 172.21.15.88:0/2820770590 wait complete. 2023-07-14T17:44:36.180 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc08c014700 1 Processor -- start 2023-07-14T17:44:36.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc08c014700 1 -- start start 2023-07-14T17:44:36.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc08c014700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0840feb30 0x7fc084069560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:36.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc08c014700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0840ffd20 0x7fc08419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:36.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc08c014700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0841006b0 0x7fc08419cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:36.181 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc08c014700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08419d3c0 con 0x7fc0840ffd20 2023-07-14T17:44:36.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc08c014700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08419d530 con 0x7fc0841006b0 2023-07-14T17:44:36.182 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc08c014700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08419d6f0 con 0x7fc0840feb30 2023-07-14T17:44:36.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc0895af700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0840ffd20 0x7fc08419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:36.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc0895af700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0840ffd20 0x7fc08419a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43480/0 (socket says 172.21.15.88:43480) 2023-07-14T17:44:36.183 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc0895af700 1 -- 172.21.15.88:0/4164337476 learned_addr learned my addr 172.21.15.88:0/4164337476 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc08a5b1700 1 --2- 172.21.15.88:0/4164337476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0841006b0 0x7fc08419cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc089db0700 1 --2- 172.21.15.88:0/4164337476 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0840feb30 0x7fc084069560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc0895af700 1 -- 172.21.15.88:0/4164337476 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0840feb30 msgr2=0x7fc084069560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:36.184 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc0895af700 1 --2- 172.21.15.88:0/4164337476 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0840feb30 0x7fc084069560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc0895af700 1 -- 172.21.15.88:0/4164337476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0841006b0 msgr2=0x7fc08419cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc0895af700 1 --2- 172.21.15.88:0/4164337476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0841006b0 0x7fc08419cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc0895af700 1 -- 172.21.15.88:0/4164337476 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0740095f0 con 0x7fc0840ffd20 2023-07-14T17:44:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc08a5b1700 1 --2- 172.21.15.88:0/4164337476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0841006b0 0x7fc08419cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:36.185 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.194+0000 7fc0895af700 1 --2- 172.21.15.88:0/4164337476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0840ffd20 0x7fc08419a5e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc074003a50 tx=0x7fc074003b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:36.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.195+0000 7fc07affd700 1 -- 172.21.15.88:0/4164337476 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07401d070 con 0x7fc0840ffd20 2023-07-14T17:44:36.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.195+0000 7fc08c014700 1 -- 172.21.15.88:0/4164337476 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08419d970 con 0x7fc0840ffd20 2023-07-14T17:44:36.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.195+0000 7fc07affd700 1 -- 172.21.15.88:0/4164337476 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc07400bd60 con 0x7fc0840ffd20 2023-07-14T17:44:36.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.195+0000 7fc07affd700 1 -- 172.21.15.88:0/4164337476 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc074005350 con 0x7fc0840ffd20 2023-07-14T17:44:36.186 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.195+0000 7fc08c014700 1 -- 172.21.15.88:0/4164337476 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08419ddc0 con 0x7fc0840ffd20 2023-07-14T17:44:36.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.196+0000 7fc08c014700 1 -- 172.21.15.88:0/4164337476 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc084105ae0 con 0x7fc0840ffd20 2023-07-14T17:44:36.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.197+0000 7fc07affd700 1 -- 172.21.15.88:0/4164337476 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc0740054b0 con 0x7fc0840ffd20 2023-07-14T17:44:36.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.197+0000 7fc07affd700 1 --2- 172.21.15.88:0/4164337476 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc0700a1150 0x7fc0700a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:36.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.197+0000 7fc07affd700 1 -- 172.21.15.88:0/4164337476 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc0740ce0e0 con 0x7fc0840ffd20 2023-07-14T17:44:36.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.197+0000 7fc089db0700 1 --2- 172.21.15.88:0/4164337476 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc0700a1150 0x7fc0700a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:36.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.198+0000 7fc089db0700 1 --2- 172.21.15.88:0/4164337476 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc0700a1150 0x7fc0700a3600 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc0840ffb80 tx=0x7fc080005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:36.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.202+0000 7fc07affd700 1 -- 172.21.15.88:0/4164337476 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc07405b780 con 0x7fc0840ffd20 2023-07-14T17:44:36.369 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.388+0000 7fc08c014700 1 -- 172.21.15.88:0/4164337476 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc0840612f0 con 0x7fc0840ffd20 2023-07-14T17:44:36.370 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.388+0000 7fc07affd700 1 -- 172.21.15.88:0/4164337476 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc074097670 con 0x7fc0840ffd20 2023-07-14T17:44:36.370 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-07-14T17:44:36.371 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.390+0000 7fc08c014700 1 -- 172.21.15.88:0/4164337476 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc0700a1150 msgr2=0x7fc0700a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:36.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.390+0000 7fc08c014700 1 --2- 172.21.15.88:0/4164337476 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc0700a1150 0x7fc0700a3600 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc0840ffb80 tx=0x7fc080005cf0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.390+0000 7fc08c014700 1 -- 172.21.15.88:0/4164337476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0840ffd20 msgr2=0x7fc08419a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:36.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.390+0000 7fc08c014700 1 --2- 172.21.15.88:0/4164337476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0840ffd20 0x7fc08419a5e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc074003a50 tx=0x7fc074003b30 comp rx=0 tx=0).stop 2023-07-14T17:44:36.372 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.390+0000 7fc08c014700 1 -- 172.21.15.88:0/4164337476 shutdown_connections 2023-07-14T17:44:36.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.390+0000 7fc08c014700 1 --2- 172.21.15.88:0/4164337476 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc0841006b0 0x7fc08419cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.390+0000 7fc08c014700 1 --2- 172.21.15.88:0/4164337476 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc0840feb30 0x7fc084069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.390+0000 7fc08c014700 1 --2- 172.21.15.88:0/4164337476 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc0700a1150 0x7fc0700a3600 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.390+0000 7fc08c014700 1 --2- 172.21.15.88:0/4164337476 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0840ffd20 0x7fc08419a5e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.373 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.390+0000 7fc08c014700 1 -- 172.21.15.88:0/4164337476 >> 172.21.15.88:0/4164337476 conn(0x7fc0840fa190 msgr2=0x7fc0840fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:36.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.390+0000 7fc08c014700 1 -- 172.21.15.88:0/4164337476 shutdown_connections 2023-07-14T17:44:36.374 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.390+0000 7fc08c014700 1 -- 172.21.15.88:0/4164337476 wait complete. 2023-07-14T17:44:36.382 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:36 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2703033925' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T17:44:36.383 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:36 smithi088 ceph-mon[127465]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-14T17:44:36.383 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:36 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2886957447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:36.384 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T17:44:36.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:36 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2703033925' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T17:44:36.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:36 smithi178 ceph-mon[129368]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-14T17:44:36.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:36 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2886957447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:36.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:36 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2703033925' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T17:44:36.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:36 smithi107 ceph-mon[128999]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-14T17:44:36.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:36 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2886957447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:36.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.526+0000 7f15226f9700 1 -- 172.21.15.88:0/455831063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f151c100690 msgr2=0x7f151c106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:36.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.526+0000 7f15226f9700 1 --2- 172.21.15.88:0/455831063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f151c100690 0x7f151c106f50 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f151400b0a0 tx=0x7f151400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.526+0000 7f15226f9700 1 -- 172.21.15.88:0/455831063 shutdown_connections 2023-07-14T17:44:36.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.526+0000 7f15226f9700 1 --2- 172.21.15.88:0/455831063 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f151c100690 0x7f151c106f50 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.526+0000 7f15226f9700 1 --2- 172.21.15.88:0/455831063 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f151c0feb10 0x7f151c0fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.526+0000 7f15226f9700 1 --2- 172.21.15.88:0/455831063 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f151c0ffd00 0x7f151c100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.526+0000 7f15226f9700 1 -- 172.21.15.88:0/455831063 >> 172.21.15.88:0/455831063 conn(0x7f151c0fa130 msgr2=0x7f151c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:36.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.526+0000 7f15226f9700 1 -- 172.21.15.88:0/455831063 shutdown_connections 2023-07-14T17:44:36.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.526+0000 7f15226f9700 1 -- 172.21.15.88:0/455831063 wait complete. 2023-07-14T17:44:36.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f15226f9700 1 Processor -- start 2023-07-14T17:44:36.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f15226f9700 1 -- start start 2023-07-14T17:44:36.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f15226f9700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f151c0feb10 0x7f151c1959b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:36.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f15226f9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f151c0ffd00 0x7f151c195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:36.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f15226f9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f151c100690 0x7f151c19cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:36.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f15226f9700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c19d530 con 0x7f151c0ffd00 2023-07-14T17:44:36.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f15226f9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c19d670 con 0x7f151c100690 2023-07-14T17:44:36.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f15226f9700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151c19d800 con 0x7f151c0feb10 2023-07-14T17:44:36.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f1513fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f151c0ffd00 0x7f151c195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:36.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f1513fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f151c0ffd00 0x7f151c195ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43494/0 (socket says 172.21.15.88:43494) 2023-07-14T17:44:36.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f1513fff700 1 -- 172.21.15.88:0/3154683907 learned_addr learned my addr 172.21.15.88:0/3154683907 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:36.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f151bfff700 1 --2- 172.21.15.88:0/3154683907 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f151c0feb10 0x7f151c1959b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:36.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f1520c96700 1 --2- 172.21.15.88:0/3154683907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f151c100690 0x7f151c19cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:36.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f1513fff700 1 -- 172.21.15.88:0/3154683907 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f151c0feb10 msgr2=0x7f151c1959b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:36.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f1513fff700 1 --2- 172.21.15.88:0/3154683907 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f151c0feb10 0x7f151c1959b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f1513fff700 1 -- 172.21.15.88:0/3154683907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f151c100690 msgr2=0x7f151c19cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:36.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f1513fff700 1 --2- 172.21.15.88:0/3154683907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f151c100690 0x7f151c19cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f1513fff700 1 -- 172.21.15.88:0/3154683907 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1514009d00 con 0x7f151c0ffd00 2023-07-14T17:44:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f151bfff700 1 --2- 172.21.15.88:0/3154683907 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f151c0feb10 0x7f151c1959b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.527+0000 7f1520c96700 1 --2- 172.21.15.88:0/3154683907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f151c100690 0x7f151c19cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.528+0000 7f1513fff700 1 --2- 172.21.15.88:0/3154683907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f151c0ffd00 0x7f151c195ef0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f150400c920 tx=0x7f150400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.528+0000 7f1519ffb700 1 -- 172.21.15.88:0/3154683907 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1504007b00 con 0x7f151c0ffd00 2023-07-14T17:44:36.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.528+0000 7f1519ffb700 1 -- 172.21.15.88:0/3154683907 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1504004cc0 con 0x7f151c0ffd00 2023-07-14T17:44:36.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.528+0000 7f15226f9700 1 -- 172.21.15.88:0/3154683907 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f151c19dae0 con 0x7f151c0ffd00 2023-07-14T17:44:36.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.528+0000 7f1519ffb700 1 -- 172.21.15.88:0/3154683907 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1504005590 con 0x7f151c0ffd00 2023-07-14T17:44:36.516 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.528+0000 7f15226f9700 1 -- 172.21.15.88:0/3154683907 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f151c19e040 con 0x7f151c0ffd00 2023-07-14T17:44:36.517 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.529+0000 7f15226f9700 1 -- 172.21.15.88:0/3154683907 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f151c10a1a0 con 0x7f151c0ffd00 2023-07-14T17:44:36.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.530+0000 7f1519ffb700 1 -- 172.21.15.88:0/3154683907 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1504004750 con 0x7f151c0ffd00 2023-07-14T17:44:36.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.530+0000 7f1519ffb700 1 --2- 172.21.15.88:0/3154683907 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f15080a11d0 0x7f15080a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:36.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.530+0000 7f1519ffb700 1 -- 172.21.15.88:0/3154683907 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f15040ccb70 con 0x7f151c0ffd00 2023-07-14T17:44:36.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.530+0000 7f151bfff700 1 --2- 172.21.15.88:0/3154683907 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f15080a11d0 0x7f15080a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:36.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.531+0000 7f151bfff700 1 --2- 172.21.15.88:0/3154683907 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f15080a11d0 0x7f15080a3680 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f150c005950 tx=0x7f150c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:36.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.533+0000 7f1519ffb700 1 -- 172.21.15.88:0/3154683907 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1504096030 con 0x7f151c0ffd00 2023-07-14T17:44:36.704 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.723+0000 7f15226f9700 1 -- 172.21.15.88:0/3154683907 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f151c19e650 con 0x7f151c0ffd00 2023-07-14T17:44:36.705 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.723+0000 7f1519ffb700 1 -- 172.21.15.88:0/3154683907 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f150401d020 con 0x7f151c0ffd00 2023-07-14T17:44:36.705 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-07-14T17:44:36.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.726+0000 7f15226f9700 1 -- 172.21.15.88:0/3154683907 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f15080a11d0 msgr2=0x7f15080a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:36.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.726+0000 7f15226f9700 1 --2- 172.21.15.88:0/3154683907 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f15080a11d0 0x7f15080a3680 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f150c005950 tx=0x7f150c0058e0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.726+0000 7f15226f9700 1 -- 172.21.15.88:0/3154683907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f151c0ffd00 msgr2=0x7f151c195ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:36.708 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.726+0000 7f15226f9700 1 --2- 172.21.15.88:0/3154683907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f151c0ffd00 0x7f151c195ef0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f150400c920 tx=0x7f150400cce0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.726+0000 7f15226f9700 1 -- 172.21.15.88:0/3154683907 shutdown_connections 2023-07-14T17:44:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.726+0000 7f15226f9700 1 --2- 172.21.15.88:0/3154683907 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f151c100690 0x7f151c19cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.726+0000 7f15226f9700 1 --2- 172.21.15.88:0/3154683907 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f151c0feb10 0x7f151c1959b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.726+0000 7f15226f9700 1 --2- 172.21.15.88:0/3154683907 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f15080a11d0 0x7f15080a3680 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.709 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.726+0000 7f15226f9700 1 --2- 172.21.15.88:0/3154683907 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f151c0ffd00 0x7f151c195ef0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.726+0000 7f15226f9700 1 -- 172.21.15.88:0/3154683907 >> 172.21.15.88:0/3154683907 conn(0x7f151c0fa130 msgr2=0x7f151c067800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:36.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.726+0000 7f15226f9700 1 -- 172.21.15.88:0/3154683907 shutdown_connections 2023-07-14T17:44:36.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.726+0000 7f15226f9700 1 -- 172.21.15.88:0/3154683907 wait complete. 2023-07-14T17:44:36.719 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T17:44:36.844 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.863+0000 7f5975ae5700 1 -- 172.21.15.88:0/3527348485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59701000b0 msgr2=0x7f59701004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:36.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.863+0000 7f5975ae5700 1 --2- 172.21.15.88:0/3527348485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59701000b0 0x7f59701004c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5964009990 tx=0x7f5964009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.863+0000 7f5975ae5700 1 -- 172.21.15.88:0/3527348485 shutdown_connections 2023-07-14T17:44:36.845 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.863+0000 7f5975ae5700 1 --2- 172.21.15.88:0/3527348485 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59700ff6b0 0x7f5970106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.863+0000 7f5975ae5700 1 --2- 172.21.15.88:0/3527348485 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59701000b0 0x7f59701004c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.863+0000 7f5975ae5700 1 --2- 172.21.15.88:0/3527348485 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59700fed00 0x7f59700ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.863+0000 7f5975ae5700 1 -- 172.21.15.88:0/3527348485 >> 172.21.15.88:0/3527348485 conn(0x7f59700f98a0 msgr2=0x7f59700fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:36.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.863+0000 7f5975ae5700 1 -- 172.21.15.88:0/3527348485 shutdown_connections 2023-07-14T17:44:36.846 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.863+0000 7f5975ae5700 1 -- 172.21.15.88:0/3527348485 wait complete. 2023-07-14T17:44:36.847 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.863+0000 7f5975ae5700 1 Processor -- start 2023-07-14T17:44:36.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.864+0000 7f5975ae5700 1 -- start start 2023-07-14T17:44:36.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.864+0000 7f5975ae5700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59700fed00 0x7f5970195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:36.849 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.864+0000 7f5975ae5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59700ff6b0 0x7f5970195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:36.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.864+0000 7f5975ae5700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59701000b0 0x7f597019cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:36.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.864+0000 7f5975ae5700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597019d3b0 con 0x7f59700ff6b0 2023-07-14T17:44:36.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.864+0000 7f5975ae5700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597019d4f0 con 0x7f59700fed00 2023-07-14T17:44:36.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.864+0000 7f5975ae5700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597019d680 con 0x7f59701000b0 2023-07-14T17:44:36.850 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.864+0000 7f596effd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59700ff6b0 0x7f5970195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:36.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.864+0000 7f596effd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59700ff6b0 0x7f5970195d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43508/0 (socket says 172.21.15.88:43508) 2023-07-14T17:44:36.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.864+0000 7f596effd700 1 -- 172.21.15.88:0/2653798798 learned_addr learned my addr 172.21.15.88:0/2653798798 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:36.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.864+0000 7f596f7fe700 1 --2- 172.21.15.88:0/2653798798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59700fed00 0x7f5970195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:36.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.864+0000 7f596ffff700 1 --2- 172.21.15.88:0/2653798798 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59701000b0 0x7f597019cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:36.851 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.864+0000 7f596effd700 1 -- 172.21.15.88:0/2653798798 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59701000b0 msgr2=0x7f597019cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:36.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.864+0000 7f596effd700 1 --2- 172.21.15.88:0/2653798798 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59701000b0 0x7f597019cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.864+0000 7f596effd700 1 -- 172.21.15.88:0/2653798798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59700fed00 msgr2=0x7f5970195830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:36.852 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.864+0000 7f596effd700 1 --2- 172.21.15.88:0/2653798798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59700fed00 0x7f5970195830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:36.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.864+0000 7f596effd700 1 -- 172.21.15.88:0/2653798798 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59640095f0 con 0x7f59700ff6b0 2023-07-14T17:44:36.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.865+0000 7f596effd700 1 --2- 172.21.15.88:0/2653798798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59700ff6b0 0x7f5970195d70 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f596000d950 tx=0x7f596000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:36.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.865+0000 7f596cff9700 1 -- 172.21.15.88:0/2653798798 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596000f890 con 0x7f59700ff6b0 2023-07-14T17:44:36.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.865+0000 7f596cff9700 1 -- 172.21.15.88:0/2653798798 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f596000fe80 con 0x7f59700ff6b0 2023-07-14T17:44:36.853 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.865+0000 7f5975ae5700 1 -- 172.21.15.88:0/2653798798 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f597019d960 con 0x7f59700ff6b0 2023-07-14T17:44:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.865+0000 7f596cff9700 1 -- 172.21.15.88:0/2653798798 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596000e3b0 con 0x7f59700ff6b0 2023-07-14T17:44:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.865+0000 7f5975ae5700 1 -- 172.21.15.88:0/2653798798 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f597019de70 con 0x7f59700ff6b0 2023-07-14T17:44:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.866+0000 7f5975ae5700 1 -- 172.21.15.88:0/2653798798 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5970105b00 con 0x7f59700ff6b0 2023-07-14T17:44:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.867+0000 7f596cff9700 1 -- 172.21.15.88:0/2653798798 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5960010460 con 0x7f59700ff6b0 2023-07-14T17:44:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.867+0000 7f596cff9700 1 --2- 172.21.15.88:0/2653798798 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f59580a1150 0x7f59580a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.867+0000 7f596f7fe700 1 --2- 172.21.15.88:0/2653798798 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f59580a1150 0x7f59580a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.867+0000 7f596cff9700 1 -- 172.21.15.88:0/2653798798 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f59600cc9f0 con 0x7f59700ff6b0 2023-07-14T17:44:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.868+0000 7f596f7fe700 1 --2- 172.21.15.88:0/2653798798 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f59580a1150 0x7f59580a3600 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f596400b5c0 tx=0x7f5964005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:36.856 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:36.871+0000 7f596cff9700 1 -- 172.21.15.88:0/2653798798 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5960095dd0 con 0x7f59700ff6b0 2023-07-14T17:44:37.037 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.056+0000 7f596cff9700 1 -- 172.21.15.88:0/2653798798 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f59600961a0 con 0x7f59700ff6b0 2023-07-14T17:44:37.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.065+0000 7f5975ae5700 1 -- 172.21.15.88:0/2653798798 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f597002caf0 con 0x7f59700ff6b0 2023-07-14T17:44:37.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.065+0000 7f596cff9700 1 -- 172.21.15.88:0/2653798798 <== mon.0 v2:172.21.15.88:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5960090d50 con 0x7f59700ff6b0 2023-07-14T17:44:37.048 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 64 2023-07-14T17:44:37.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.067+0000 7f5975ae5700 1 -- 172.21.15.88:0/2653798798 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f59580a1150 msgr2=0x7f59580a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:37.050 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.067+0000 7f5975ae5700 1 --2- 172.21.15.88:0/2653798798 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f59580a1150 0x7f59580a3600 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f596400b5c0 tx=0x7f5964005fb0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.067+0000 7f5975ae5700 1 -- 172.21.15.88:0/2653798798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59700ff6b0 msgr2=0x7f5970195d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:37.051 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.067+0000 7f5975ae5700 1 --2- 172.21.15.88:0/2653798798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59700ff6b0 0x7f5970195d70 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f596000d950 tx=0x7f596000dc60 comp rx=0 tx=0).stop 2023-07-14T17:44:37.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.067+0000 7f5975ae5700 1 -- 172.21.15.88:0/2653798798 shutdown_connections 2023-07-14T17:44:37.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.067+0000 7f5975ae5700 1 --2- 172.21.15.88:0/2653798798 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f59701000b0 0x7f597019cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.052 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.067+0000 7f5975ae5700 1 --2- 172.21.15.88:0/2653798798 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f59700fed00 0x7f5970195830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.067+0000 7f5975ae5700 1 --2- 172.21.15.88:0/2653798798 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f59580a1150 0x7f59580a3600 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.053 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.067+0000 7f5975ae5700 1 --2- 172.21.15.88:0/2653798798 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59700ff6b0 0x7f5970195d70 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.067+0000 7f5975ae5700 1 -- 172.21.15.88:0/2653798798 >> 172.21.15.88:0/2653798798 conn(0x7f59700f98a0 msgr2=0x7f59700fbbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:37.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.067+0000 7f5975ae5700 1 -- 172.21.15.88:0/2653798798 shutdown_connections 2023-07-14T17:44:37.054 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.067+0000 7f5975ae5700 1 -- 172.21.15.88:0/2653798798 wait complete. 2023-07-14T17:44:37.061 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T17:44:37.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.204+0000 7f1f2252f700 1 -- 172.21.15.88:0/585321006 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f1c1018e0 msgr2=0x7f1f1c1081a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:37.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.204+0000 7f1f2252f700 1 --2- 172.21.15.88:0/585321006 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f1c1018e0 0x7f1f1c1081a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1f1400b0a0 tx=0x7f1f1400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.204+0000 7f1f2252f700 1 -- 172.21.15.88:0/585321006 shutdown_connections 2023-07-14T17:44:37.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.204+0000 7f1f2252f700 1 --2- 172.21.15.88:0/585321006 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f1c1018e0 0x7f1f1c1081a0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.204+0000 7f1f2252f700 1 --2- 172.21.15.88:0/585321006 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f1c0fdee0 0x7f1f1c0fe2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.204+0000 7f1f2252f700 1 --2- 172.21.15.88:0/585321006 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f1c100f50 0x7f1f1c1013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.188 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.204+0000 7f1f2252f700 1 -- 172.21.15.88:0/585321006 >> 172.21.15.88:0/585321006 conn(0x7f1f1c0f98a0 msgr2=0x7f1f1c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:37.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.204+0000 7f1f2252f700 1 -- 172.21.15.88:0/585321006 shutdown_connections 2023-07-14T17:44:37.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.204+0000 7f1f2252f700 1 -- 172.21.15.88:0/585321006 wait complete. 2023-07-14T17:44:37.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.204+0000 7f1f2252f700 1 Processor -- start 2023-07-14T17:44:37.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.204+0000 7f1f2252f700 1 -- start start 2023-07-14T17:44:37.189 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.204+0000 7f1f2252f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f1c0fdee0 0x7f1f1c195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:37.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f2252f700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f1c100f50 0x7f1f1c195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:37.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f2252f700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f1c1018e0 0x7f1f1c19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:37.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f2252f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c19d4c0 con 0x7f1f1c0fdee0 2023-07-14T17:44:37.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f2252f700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c19d600 con 0x7f1f1c1018e0 2023-07-14T17:44:37.190 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f2252f700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c19d790 con 0x7f1f1c100f50 2023-07-14T17:44:37.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f1bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f1c0fdee0 0x7f1f1c195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:37.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f1bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f1c0fdee0 0x7f1f1c195940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43520/0 (socket says 172.21.15.88:43520) 2023-07-14T17:44:37.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f1bfff700 1 -- 172.21.15.88:0/1177088126 learned_addr learned my addr 172.21.15.88:0/1177088126 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:37.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f13fff700 1 --2- 172.21.15.88:0/1177088126 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f1c100f50 0x7f1f1c195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:37.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f20acc700 1 --2- 172.21.15.88:0/1177088126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f1c1018e0 0x7f1f1c19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:37.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f1bfff700 1 -- 172.21.15.88:0/1177088126 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f1c100f50 msgr2=0x7f1f1c195e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:37.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f1bfff700 1 --2- 172.21.15.88:0/1177088126 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f1c100f50 0x7f1f1c195e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f1bfff700 1 -- 172.21.15.88:0/1177088126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f1c1018e0 msgr2=0x7f1f1c19cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:37.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f1bfff700 1 --2- 172.21.15.88:0/1177088126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f1c1018e0 0x7f1f1c19cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f1bfff700 1 -- 172.21.15.88:0/1177088126 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f14009d00 con 0x7f1f1c0fdee0 2023-07-14T17:44:37.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f1bfff700 1 --2- 172.21.15.88:0/1177088126 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f1c0fdee0 0x7f1f1c195940 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c00d980 tx=0x7f1f0c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:37.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f19ffb700 1 -- 172.21.15.88:0/1177088126 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c004020 con 0x7f1f1c0fdee0 2023-07-14T17:44:37.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f19ffb700 1 -- 172.21.15.88:0/1177088126 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1f0c009730 con 0x7f1f1c0fdee0 2023-07-14T17:44:37.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f2252f700 1 -- 172.21.15.88:0/1177088126 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1c19da70 con 0x7f1f1c0fdee0 2023-07-14T17:44:37.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f19ffb700 1 -- 172.21.15.88:0/1177088126 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c003a00 con 0x7f1f1c0fdee0 2023-07-14T17:44:37.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.205+0000 7f1f2252f700 1 -- 172.21.15.88:0/1177088126 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1c19dfd0 con 0x7f1f1c0fdee0 2023-07-14T17:44:37.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.206+0000 7f1f2252f700 1 -- 172.21.15.88:0/1177088126 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1c1072b0 con 0x7f1f1c0fdee0 2023-07-14T17:44:37.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.207+0000 7f1f19ffb700 1 -- 172.21.15.88:0/1177088126 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f1f0c0098a0 con 0x7f1f1c0fdee0 2023-07-14T17:44:37.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.208+0000 7f1f19ffb700 1 --2- 172.21.15.88:0/1177088126 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1f080a12e0 0x7f1f080a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:37.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.208+0000 7f1f13fff700 1 --2- 172.21.15.88:0/1177088126 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1f080a12e0 0x7f1f080a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:37.195 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.208+0000 7f1f19ffb700 1 -- 172.21.15.88:0/1177088126 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1f0c0ccdf0 con 0x7f1f1c0fdee0 2023-07-14T17:44:37.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.208+0000 7f1f13fff700 1 --2- 172.21.15.88:0/1177088126 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1f080a12e0 0x7f1f080a3790 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400ac50 tx=0x7f1f0400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:37.196 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.211+0000 7f1f19ffb700 1 -- 172.21.15.88:0/1177088126 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1f0c0960f0 con 0x7f1f1c0fdee0 2023-07-14T17:44:37.381 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.400+0000 7f1f2252f700 1 -- 172.21.15.88:0/1177088126 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1f1c02cb50 con 0x7f1f1c0fdee0 2023-07-14T17:44:37.382 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.401+0000 7f1f19ffb700 1 -- 172.21.15.88:0/1177088126 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1f0c099710 con 0x7f1f1c0fdee0 2023-07-14T17:44:37.383 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-07-14T17:44:37.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.403+0000 7f1f2252f700 1 -- 172.21.15.88:0/1177088126 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1f080a12e0 msgr2=0x7f1f080a3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:37.384 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.403+0000 7f1f2252f700 1 --2- 172.21.15.88:0/1177088126 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1f080a12e0 0x7f1f080a3790 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1f0400ac50 tx=0x7f1f0400a380 comp rx=0 tx=0).stop 2023-07-14T17:44:37.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.403+0000 7f1f2252f700 1 -- 172.21.15.88:0/1177088126 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f1c0fdee0 msgr2=0x7f1f1c195940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:37.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.403+0000 7f1f2252f700 1 --2- 172.21.15.88:0/1177088126 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f1c0fdee0 0x7f1f1c195940 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c00d980 tx=0x7f1f0c00dc90 comp rx=0 tx=0).stop 2023-07-14T17:44:37.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.403+0000 7f1f2252f700 1 -- 172.21.15.88:0/1177088126 shutdown_connections 2023-07-14T17:44:37.385 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.403+0000 7f1f2252f700 1 --2- 172.21.15.88:0/1177088126 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f1f1c1018e0 0x7f1f1c19cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.403+0000 7f1f2252f700 1 --2- 172.21.15.88:0/1177088126 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f1f080a12e0 0x7f1f080a3790 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.403+0000 7f1f2252f700 1 --2- 172.21.15.88:0/1177088126 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1f1c0fdee0 0x7f1f1c195940 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.403+0000 7f1f2252f700 1 --2- 172.21.15.88:0/1177088126 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f1f1c100f50 0x7f1f1c195e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.403+0000 7f1f2252f700 1 -- 172.21.15.88:0/1177088126 >> 172.21.15.88:0/1177088126 conn(0x7f1f1c0f98a0 msgr2=0x7f1f1c1043f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:37.386 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.403+0000 7f1f2252f700 1 -- 172.21.15.88:0/1177088126 shutdown_connections 2023-07-14T17:44:37.387 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.403+0000 7f1f2252f700 1 -- 172.21.15.88:0/1177088126 wait complete. 2023-07-14T17:44:37.395 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:37 smithi088 ceph-mon[127465]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:44:37.395 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:37 smithi088 ceph-mon[127465]: pgmap v314: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 767 B/s wr, 12 op/s 2023-07-14T17:44:37.395 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:37 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/4164337476' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:44:37.396 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:37 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3154683907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:44:37.397 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T17:44:37.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:37 smithi178 ceph-mon[129368]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:44:37.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:37 smithi178 ceph-mon[129368]: pgmap v314: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 767 B/s wr, 12 op/s 2023-07-14T17:44:37.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:37 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/4164337476' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:44:37.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:37 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3154683907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:44:37.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:37 smithi107 ceph-mon[128999]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T17:44:37.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:37 smithi107 ceph-mon[128999]: pgmap v314: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 767 B/s wr, 12 op/s 2023-07-14T17:44:37.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:37 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/4164337476' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:44:37.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:37 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3154683907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:44:37.518 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.537+0000 7fe005971700 1 -- 172.21.15.88:0/2399484848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe000100e00 msgr2=0x7fe000101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:37.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.537+0000 7fe005971700 1 --2- 172.21.15.88:0/2399484848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe000100e00 0x7fe000101250 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fdff0009990 tx=0x7fdff0009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.537+0000 7fe005971700 1 -- 172.21.15.88:0/2399484848 shutdown_connections 2023-07-14T17:44:37.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.537+0000 7fe005971700 1 --2- 172.21.15.88:0/2399484848 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe000101790 0x7fe000108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.519 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.537+0000 7fe005971700 1 --2- 172.21.15.88:0/2399484848 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe0000ffc10 0x7fe000100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.537+0000 7fe005971700 1 --2- 172.21.15.88:0/2399484848 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe000100e00 0x7fe000101250 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.537+0000 7fe005971700 1 -- 172.21.15.88:0/2399484848 >> 172.21.15.88:0/2399484848 conn(0x7fe0000fb250 msgr2=0x7fe0000fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:37.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.537+0000 7fe005971700 1 -- 172.21.15.88:0/2399484848 shutdown_connections 2023-07-14T17:44:37.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.537+0000 7fe005971700 1 -- 172.21.15.88:0/2399484848 wait complete. 2023-07-14T17:44:37.520 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.537+0000 7fe005971700 1 Processor -- start 2023-07-14T17:44:37.522 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fe005971700 1 -- start start 2023-07-14T17:44:37.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fe005971700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0000ffc10 0x7fe0000743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:37.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fe005971700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe000100e00 0x7fe000072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:37.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fe005971700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe000101790 0x7fe000072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:37.523 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fe005971700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe000073710 con 0x7fe000101790 2023-07-14T17:44:37.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fe005971700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe000073880 con 0x7fe000100e00 2023-07-14T17:44:37.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fe005971700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00019f880 con 0x7fe0000ffc10 2023-07-14T17:44:37.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fdfff7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe000101790 0x7fe000072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:37.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fdfff7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe000101790 0x7fe000072f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43538/0 (socket says 172.21.15.88:43538) 2023-07-14T17:44:37.524 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fdffeffd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0000ffc10 0x7fe0000743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:37.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fdfff7fe700 1 -- 172.21.15.88:0/876869901 learned_addr learned my addr 172.21.15.88:0/876869901 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:37.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fdffe7fc700 1 --2- 172.21.15.88:0/876869901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe000100e00 0x7fe000072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:37.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fdfff7fe700 1 -- 172.21.15.88:0/876869901 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0000ffc10 msgr2=0x7fe0000743f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:37.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fdfff7fe700 1 --2- 172.21.15.88:0/876869901 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0000ffc10 0x7fe0000743f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.525 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fdfff7fe700 1 -- 172.21.15.88:0/876869901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe000100e00 msgr2=0x7fe000072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:37.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fdfff7fe700 1 --2- 172.21.15.88:0/876869901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe000100e00 0x7fe000072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fdfff7fe700 1 -- 172.21.15.88:0/876869901 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff00095f0 con 0x7fe000101790 2023-07-14T17:44:37.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fdfff7fe700 1 --2- 172.21.15.88:0/876869901 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe000101790 0x7fe000072f70 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fdff800ada0 tx=0x7fdff800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:37.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.538+0000 7fe00496f700 1 -- 172.21.15.88:0/876869901 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff800f070 con 0x7fe000101790 2023-07-14T17:44:37.526 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.539+0000 7fe005971700 1 -- 172.21.15.88:0/876869901 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe00019fb60 con 0x7fe000101790 2023-07-14T17:44:37.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.539+0000 7fe00496f700 1 -- 172.21.15.88:0/876869901 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdff8007b30 con 0x7fe000101790 2023-07-14T17:44:37.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.539+0000 7fe00496f700 1 -- 172.21.15.88:0/876869901 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff8005280 con 0x7fe000101790 2023-07-14T17:44:37.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.539+0000 7fe005971700 1 -- 172.21.15.88:0/876869901 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0001a0020 con 0x7fe000101790 2023-07-14T17:44:37.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.540+0000 7fe005971700 1 -- 172.21.15.88:0/876869901 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe00010b290 con 0x7fe000101790 2023-07-14T17:44:37.527 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.540+0000 7fe00496f700 1 -- 172.21.15.88:0/876869901 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fdff800c980 con 0x7fe000101790 2023-07-14T17:44:37.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.541+0000 7fe00496f700 1 --2- 172.21.15.88:0/876869901 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fdfe80a11e0 0x7fdfe80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:37.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.541+0000 7fdffeffd700 1 --2- 172.21.15.88:0/876869901 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fdfe80a11e0 0x7fdfe80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:37.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.541+0000 7fe00496f700 1 -- 172.21.15.88:0/876869901 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdff80cc440 con 0x7fe000101790 2023-07-14T17:44:37.528 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.541+0000 7fdffeffd700 1 --2- 172.21.15.88:0/876869901 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fdfe80a11e0 0x7fdfe80a3690 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe000100c60 tx=0x7fdff400a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:37.529 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.545+0000 7fe00496f700 1 -- 172.21.15.88:0/876869901 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdff80957b0 con 0x7fe000101790 2023-07-14T17:44:37.729 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.748+0000 7fe005971700 1 -- 172.21.15.88:0/876869901 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe000106b60 con 0x7fe000101790 2023-07-14T17:44:37.730 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.748+0000 7fe00496f700 1 -- 172.21.15.88:0/876869901 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) v1 ==== 81+0+591 (secure 0 0 0) 0x7fdff800cc30 con 0x7fe000101790 2023-07-14T17:44:37.730 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/1441439704 2023-07-15T17:36:43.799741+0000 2023-07-14T17:44:37.730 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/3897985004 2023-07-15T17:36:43.799741+0000 2023-07-14T17:44:37.731 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6801/2424184187 2023-07-15T17:36:43.799741+0000 2023-07-14T17:44:37.731 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6801/1369181448 2023-07-15T17:36:21.153428+0000 2023-07-14T17:44:37.731 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6800/1369181448 2023-07-15T17:36:21.153428+0000 2023-07-14T17:44:37.731 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/1790623516 2023-07-15T17:36:21.153428+0000 2023-07-14T17:44:37.731 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/3232411191 2023-07-15T17:36:21.153428+0000 2023-07-14T17:44:37.732 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/3329866608 2023-07-15T17:36:21.153428+0000 2023-07-14T17:44:37.732 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/954461155 2023-07-15T17:36:43.799741+0000 2023-07-14T17:44:37.732 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6800/2424184187 2023-07-15T17:36:43.799741+0000 2023-07-14T17:44:37.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.750+0000 7fe005971700 1 -- 172.21.15.88:0/876869901 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fdfe80a11e0 msgr2=0x7fdfe80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:37.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.750+0000 7fe005971700 1 --2- 172.21.15.88:0/876869901 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fdfe80a11e0 0x7fdfe80a3690 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe000100c60 tx=0x7fdff400a3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.733 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.750+0000 7fe005971700 1 -- 172.21.15.88:0/876869901 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe000101790 msgr2=0x7fe000072f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:37.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.750+0000 7fe005971700 1 --2- 172.21.15.88:0/876869901 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe000101790 0x7fe000072f70 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fdff800ada0 tx=0x7fdff800c330 comp rx=0 tx=0).stop 2023-07-14T17:44:37.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.750+0000 7fe005971700 1 -- 172.21.15.88:0/876869901 shutdown_connections 2023-07-14T17:44:37.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.750+0000 7fe005971700 1 --2- 172.21.15.88:0/876869901 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fdfe80a11e0 0x7fdfe80a3690 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.750+0000 7fe005971700 1 --2- 172.21.15.88:0/876869901 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe000101790 0x7fe000072f70 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.750+0000 7fe005971700 1 --2- 172.21.15.88:0/876869901 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fe0000ffc10 0x7fe0000743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.750+0000 7fe005971700 1 --2- 172.21.15.88:0/876869901 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fe000100e00 0x7fe000072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.750+0000 7fe005971700 1 -- 172.21.15.88:0/876869901 >> 172.21.15.88:0/876869901 conn(0x7fe0000fb250 msgr2=0x7fe0000fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:37.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.750+0000 7fe005971700 1 -- 172.21.15.88:0/876869901 shutdown_connections 2023-07-14T17:44:37.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.750+0000 7fe005971700 1 -- 172.21.15.88:0/876869901 wait complete. 2023-07-14T17:44:37.735 INFO:teuthology.orchestra.run.smithi088.stderr:listed 10 entries 2023-07-14T17:44:37.744 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T17:44:37.744 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T17:44:37.762 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:44:37.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.942+0000 7ff67a87c700 1 -- 172.21.15.88:0/1750707097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6740feb00 msgr2=0x7ff6740fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:37.924 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.942+0000 7ff67a87c700 1 --2- 172.21.15.88:0/1750707097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6740feb00 0x7ff6740fef10 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff664009990 tx=0x7ff664009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.942+0000 7ff67a87c700 1 -- 172.21.15.88:0/1750707097 shutdown_connections 2023-07-14T17:44:37.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.942+0000 7ff67a87c700 1 --2- 172.21.15.88:0/1750707097 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff674100680 0x7ff674106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.925 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.942+0000 7ff67a87c700 1 --2- 172.21.15.88:0/1750707097 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6740feb00 0x7ff6740fef10 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.942+0000 7ff67a87c700 1 --2- 172.21.15.88:0/1750707097 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6740ffcf0 0x7ff674100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.942+0000 7ff67a87c700 1 -- 172.21.15.88:0/1750707097 >> 172.21.15.88:0/1750707097 conn(0x7ff6740fa100 msgr2=0x7ff6740fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:37.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.942+0000 7ff67a87c700 1 -- 172.21.15.88:0/1750707097 shutdown_connections 2023-07-14T17:44:37.926 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.943+0000 7ff67a87c700 1 -- 172.21.15.88:0/1750707097 wait complete. 2023-07-14T17:44:37.928 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.943+0000 7ff67a87c700 1 Processor -- start 2023-07-14T17:44:37.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.943+0000 7ff67a87c700 1 -- start start 2023-07-14T17:44:37.929 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff67a87c700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6740feb00 0x7ff674193680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:37.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff67a87c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6740ffcf0 0x7ff674193bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:37.930 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff67a87c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff674100680 0x7ff67419ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:37.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff67a87c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67419b200 con 0x7ff6740ffcf0 2023-07-14T17:44:37.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff67a87c700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67419b340 con 0x7ff674100680 2023-07-14T17:44:37.931 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff67a87c700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67419b4d0 con 0x7ff6740feb00 2023-07-14T17:44:37.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff6737fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6740ffcf0 0x7ff674193bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:37.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff6737fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6740ffcf0 0x7ff674193bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43550/0 (socket says 172.21.15.88:43550) 2023-07-14T17:44:37.932 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff6737fe700 1 -- 172.21.15.88:0/3310910981 learned_addr learned my addr 172.21.15.88:0/3310910981 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:37.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff673fff700 1 --2- 172.21.15.88:0/3310910981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6740feb00 0x7ff674193680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:37.933 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff678e19700 1 --2- 172.21.15.88:0/3310910981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff674100680 0x7ff67419ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff6737fe700 1 -- 172.21.15.88:0/3310910981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6740feb00 msgr2=0x7ff674193680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:37.934 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff6737fe700 1 --2- 172.21.15.88:0/3310910981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6740feb00 0x7ff674193680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff6737fe700 1 -- 172.21.15.88:0/3310910981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff674100680 msgr2=0x7ff67419ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff6737fe700 1 --2- 172.21.15.88:0/3310910981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff674100680 0x7ff67419ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:37.935 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff6737fe700 1 -- 172.21.15.88:0/3310910981 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6640095f0 con 0x7ff6740ffcf0 2023-07-14T17:44:37.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff678e19700 1 --2- 172.21.15.88:0/3310910981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff674100680 0x7ff67419ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:37.936 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff6737fe700 1 --2- 172.21.15.88:0/3310910981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6740ffcf0 0x7ff674193bc0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff66800b530 tx=0x7ff66800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:37.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff6717fa700 1 -- 172.21.15.88:0/3310910981 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff668011830 con 0x7ff6740ffcf0 2023-07-14T17:44:37.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff67a87c700 1 -- 172.21.15.88:0/3310910981 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff67419b7b0 con 0x7ff6740ffcf0 2023-07-14T17:44:37.937 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff6717fa700 1 -- 172.21.15.88:0/3310910981 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff668011e20 con 0x7ff6740ffcf0 2023-07-14T17:44:37.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.944+0000 7ff6717fa700 1 -- 172.21.15.88:0/3310910981 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66800f2e0 con 0x7ff6740ffcf0 2023-07-14T17:44:37.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.945+0000 7ff67a87c700 1 -- 172.21.15.88:0/3310910981 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67419bd90 con 0x7ff6740ffcf0 2023-07-14T17:44:37.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.945+0000 7ff67a87c700 1 -- 172.21.15.88:0/3310910981 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff674105a80 con 0x7ff6740ffcf0 2023-07-14T17:44:37.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.946+0000 7ff6717fa700 1 -- 172.21.15.88:0/3310910981 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff668011990 con 0x7ff6740ffcf0 2023-07-14T17:44:37.939 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.947+0000 7ff6717fa700 1 --2- 172.21.15.88:0/3310910981 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff65c0a3430 0x7ff65c0a58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:37.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.947+0000 7ff673fff700 1 --2- 172.21.15.88:0/3310910981 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff65c0a3430 0x7ff65c0a58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:37.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.947+0000 7ff6717fa700 1 -- 172.21.15.88:0/3310910981 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff6680cca10 con 0x7ff6740ffcf0 2023-07-14T17:44:37.940 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.947+0000 7ff673fff700 1 --2- 172.21.15.88:0/3310910981 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff65c0a3430 0x7ff65c0a58e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff66400b5c0 tx=0x7ff664005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:37.941 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:37.950+0000 7ff6717fa700 1 -- 172.21.15.88:0/3310910981 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff668095da0 con 0x7ff6740ffcf0 2023-07-14T17:44:38.124 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.143+0000 7ff67a87c700 1 -- 172.21.15.88:0/3310910981 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff67402cae0 con 0x7ff6740ffcf0 2023-07-14T17:44:38.126 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.144+0000 7ff6717fa700 1 -- 172.21.15.88:0/3310910981 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24512 (secure 0 0 0) 0x7ff66804dfd0 con 0x7ff6740ffcf0 2023-07-14T17:44:38.126 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:38.127 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":88,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:43:44.887959+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:44:38.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.146+0000 7ff67a87c700 1 -- 172.21.15.88:0/3310910981 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff65c0a3430 msgr2=0x7ff65c0a58e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:38.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.146+0000 7ff67a87c700 1 --2- 172.21.15.88:0/3310910981 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff65c0a3430 0x7ff65c0a58e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff66400b5c0 tx=0x7ff664005960 comp rx=0 tx=0).stop 2023-07-14T17:44:38.128 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.146+0000 7ff67a87c700 1 -- 172.21.15.88:0/3310910981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6740ffcf0 msgr2=0x7ff674193bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:38.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.146+0000 7ff67a87c700 1 --2- 172.21.15.88:0/3310910981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6740ffcf0 0x7ff674193bc0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff66800b530 tx=0x7ff66800b840 comp rx=0 tx=0).stop 2023-07-14T17:44:38.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.147+0000 7ff67a87c700 1 -- 172.21.15.88:0/3310910981 shutdown_connections 2023-07-14T17:44:38.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.147+0000 7ff67a87c700 1 --2- 172.21.15.88:0/3310910981 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7ff674100680 0x7ff67419ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.147+0000 7ff67a87c700 1 --2- 172.21.15.88:0/3310910981 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7ff6740feb00 0x7ff674193680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.129 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.147+0000 7ff67a87c700 1 --2- 172.21.15.88:0/3310910981 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7ff65c0a3430 0x7ff65c0a58e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.147+0000 7ff67a87c700 1 --2- 172.21.15.88:0/3310910981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6740ffcf0 0x7ff674193bc0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.147+0000 7ff67a87c700 1 -- 172.21.15.88:0/3310910981 >> 172.21.15.88:0/3310910981 conn(0x7ff6740fa100 msgr2=0x7ff6740fc410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:38.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.147+0000 7ff67a87c700 1 -- 172.21.15.88:0/3310910981 shutdown_connections 2023-07-14T17:44:38.130 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.147+0000 7ff67a87c700 1 -- 172.21.15.88:0/3310910981 wait complete. 2023-07-14T17:44:38.141 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T17:38:50.126535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T17:42:40.725870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T17:42:41.512749+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T17:42:44.764573+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T17:44:38.142 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T17:44:38.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.286+0000 7fd4e94e7700 1 -- 172.21.15.88:0/3215341247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4e40fe2c0 msgr2=0x7fd4e40fe6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:38.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.286+0000 7fd4e94e7700 1 --2- 172.21.15.88:0/3215341247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4e40fe2c0 0x7fd4e40fe6d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400b0a0 tx=0x7fd4d400b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.286+0000 7fd4e94e7700 1 -- 172.21.15.88:0/3215341247 shutdown_connections 2023-07-14T17:44:38.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.286+0000 7fd4e94e7700 1 --2- 172.21.15.88:0/3215341247 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4e40ffe40 0x7fd4e4106700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.286+0000 7fd4e94e7700 1 --2- 172.21.15.88:0/3215341247 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4e40fe2c0 0x7fd4e40fe6d0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.286+0000 7fd4e94e7700 1 --2- 172.21.15.88:0/3215341247 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4e40ff4b0 0x7fd4e40ff900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.286+0000 7fd4e94e7700 1 -- 172.21.15.88:0/3215341247 >> 172.21.15.88:0/3215341247 conn(0x7fd4e40f98e0 msgr2=0x7fd4e40fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:38.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.286+0000 7fd4e94e7700 1 -- 172.21.15.88:0/3215341247 shutdown_connections 2023-07-14T17:44:38.269 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.286+0000 7fd4e94e7700 1 -- 172.21.15.88:0/3215341247 wait complete. 2023-07-14T17:44:38.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.287+0000 7fd4e94e7700 1 Processor -- start 2023-07-14T17:44:38.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.287+0000 7fd4e94e7700 1 -- start start 2023-07-14T17:44:38.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.287+0000 7fd4e94e7700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4e40fe2c0 0x7fd4e4194a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:38.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.287+0000 7fd4e94e7700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4e40ff4b0 0x7fd4e4194f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:38.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.287+0000 7fd4e94e7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4e40ffe40 0x7fd4e419bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:38.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.287+0000 7fd4e94e7700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e419c5b0 con 0x7fd4e40ffe40 2023-07-14T17:44:38.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.287+0000 7fd4e94e7700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e419c6f0 con 0x7fd4e40ff4b0 2023-07-14T17:44:38.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.287+0000 7fd4e94e7700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e419c880 con 0x7fd4e40fe2c0 2023-07-14T17:44:38.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.287+0000 7fd4e37fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4e40ffe40 0x7fd4e419bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:38.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.287+0000 7fd4e37fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4e40ffe40 0x7fd4e419bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43566/0 (socket says 172.21.15.88:43566) 2023-07-14T17:44:38.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.287+0000 7fd4e37fe700 1 -- 172.21.15.88:0/1325378940 learned_addr learned my addr 172.21.15.88:0/1325378940 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:38.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.287+0000 7fd4e27fc700 1 --2- 172.21.15.88:0/1325378940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4e40ff4b0 0x7fd4e4194f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:38.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.288+0000 7fd4e2ffd700 1 --2- 172.21.15.88:0/1325378940 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4e40fe2c0 0x7fd4e4194a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:38.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.288+0000 7fd4e37fe700 1 -- 172.21.15.88:0/1325378940 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4e40fe2c0 msgr2=0x7fd4e4194a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:38.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.288+0000 7fd4e37fe700 1 --2- 172.21.15.88:0/1325378940 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4e40fe2c0 0x7fd4e4194a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.288+0000 7fd4e37fe700 1 -- 172.21.15.88:0/1325378940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4e40ff4b0 msgr2=0x7fd4e4194f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:38.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.288+0000 7fd4e37fe700 1 --2- 172.21.15.88:0/1325378940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4e40ff4b0 0x7fd4e4194f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.288+0000 7fd4e37fe700 1 -- 172.21.15.88:0/1325378940 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d4009d00 con 0x7fd4e40ffe40 2023-07-14T17:44:38.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.288+0000 7fd4e37fe700 1 --2- 172.21.15.88:0/1325378940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4e40ffe40 0x7fd4e419bfe0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00d980 tx=0x7fd4dc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:38.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.288+0000 7fd4dbfff700 1 -- 172.21.15.88:0/1325378940 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc004220 con 0x7fd4e40ffe40 2023-07-14T17:44:38.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.288+0000 7fd4e94e7700 1 -- 172.21.15.88:0/1325378940 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e419cb60 con 0x7fd4e40ffe40 2023-07-14T17:44:38.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.288+0000 7fd4dbfff700 1 -- 172.21.15.88:0/1325378940 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd4dc0075e0 con 0x7fd4e40ffe40 2023-07-14T17:44:38.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.288+0000 7fd4dbfff700 1 -- 172.21.15.88:0/1325378940 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc005570 con 0x7fd4e40ffe40 2023-07-14T17:44:38.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.288+0000 7fd4e94e7700 1 -- 172.21.15.88:0/1325378940 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e419d020 con 0x7fd4e40ffe40 2023-07-14T17:44:38.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.289+0000 7fd4e94e7700 1 -- 172.21.15.88:0/1325378940 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e4104e90 con 0x7fd4e40ffe40 2023-07-14T17:44:38.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.290+0000 7fd4dbfff700 1 -- 172.21.15.88:0/1325378940 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd4dc004750 con 0x7fd4e40ffe40 2023-07-14T17:44:38.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.290+0000 7fd4dbfff700 1 --2- 172.21.15.88:0/1325378940 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd4d00a11e0 0x7fd4d00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:38.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.291+0000 7fd4e2ffd700 1 --2- 172.21.15.88:0/1325378940 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd4d00a11e0 0x7fd4d00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:38.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.291+0000 7fd4dbfff700 1 -- 172.21.15.88:0/1325378940 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd4dc0cd0c0 con 0x7fd4e40ffe40 2023-07-14T17:44:38.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.291+0000 7fd4e2ffd700 1 --2- 172.21.15.88:0/1325378940 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd4d00a11e0 0x7fd4d00a3690 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400b830 tx=0x7fd4d4009be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:38.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.294+0000 7fd4dbfff700 1 -- 172.21.15.88:0/1325378940 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd4dc0963a0 con 0x7fd4e40ffe40 2023-07-14T17:44:38.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:38 smithi178 ceph-mon[129368]: mgrmap e19: x(active, since 7m), standbys: z, y 2023-07-14T17:44:38.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:38 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2653798798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:44:38.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:38 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1177088126' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:44:38.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:38 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/876869901' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T17:44:38.440 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:38 smithi088 ceph-mon[127465]: mgrmap e19: x(active, since 7m), standbys: z, y 2023-07-14T17:44:38.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:38 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2653798798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:44:38.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:38 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1177088126' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:44:38.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:38 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/876869901' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T17:44:38.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.485+0000 7fd4e94e7700 1 -- 172.21.15.88:0/1325378940 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd4e4061250 con 0x7fd4e40ffe40 2023-07-14T17:44:38.466 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.485+0000 7fd4dbfff700 1 -- 172.21.15.88:0/1325378940 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd4dc0160a0 con 0x7fd4e40ffe40 2023-07-14T17:44:38.467 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 1 2023-07-14T17:44:38.468 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.487+0000 7fd4e94e7700 1 -- 172.21.15.88:0/1325378940 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd4d00a11e0 msgr2=0x7fd4d00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:38.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.487+0000 7fd4e94e7700 1 --2- 172.21.15.88:0/1325378940 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd4d00a11e0 0x7fd4d00a3690 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd4d400b830 tx=0x7fd4d4009be0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.487+0000 7fd4e94e7700 1 -- 172.21.15.88:0/1325378940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4e40ffe40 msgr2=0x7fd4e419bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:38.469 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.487+0000 7fd4e94e7700 1 --2- 172.21.15.88:0/1325378940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4e40ffe40 0x7fd4e419bfe0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00d980 tx=0x7fd4dc00dc90 comp rx=0 tx=0).stop 2023-07-14T17:44:38.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.487+0000 7fd4e94e7700 1 -- 172.21.15.88:0/1325378940 shutdown_connections 2023-07-14T17:44:38.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.487+0000 7fd4e94e7700 1 --2- 172.21.15.88:0/1325378940 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fd4d00a11e0 0x7fd4d00a3690 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.487+0000 7fd4e94e7700 1 --2- 172.21.15.88:0/1325378940 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd4e40ffe40 0x7fd4e419bfe0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.487+0000 7fd4e94e7700 1 --2- 172.21.15.88:0/1325378940 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fd4e40fe2c0 0x7fd4e4194a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.470 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.487+0000 7fd4e94e7700 1 --2- 172.21.15.88:0/1325378940 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fd4e40ff4b0 0x7fd4e4194f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.487+0000 7fd4e94e7700 1 -- 172.21.15.88:0/1325378940 >> 172.21.15.88:0/1325378940 conn(0x7fd4e40f98e0 msgr2=0x7fd4e40fb480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:38.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.487+0000 7fd4e94e7700 1 -- 172.21.15.88:0/1325378940 shutdown_connections 2023-07-14T17:44:38.471 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.487+0000 7fd4e94e7700 1 -- 172.21.15.88:0/1325378940 wait complete. 2023-07-14T17:44:38.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:38 smithi107 ceph-mon[128999]: mgrmap e19: x(active, since 7m), standbys: z, y 2023-07-14T17:44:38.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:38 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2653798798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:44:38.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:38 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1177088126' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:44:38.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:38 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/876869901' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T17:44:38.481 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T17:44:38.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.626+0000 7f33a2204700 1 -- 172.21.15.88:0/3036354138 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f339c069690 msgr2=0x7f339c1027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:38.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.626+0000 7f33a2204700 1 --2- 172.21.15.88:0/3036354138 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f339c069690 0x7f339c1027c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3390009990 tx=0x7f3390009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.608 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.626+0000 7f33a2204700 1 -- 172.21.15.88:0/3036354138 shutdown_connections 2023-07-14T17:44:38.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.626+0000 7f33a2204700 1 --2- 172.21.15.88:0/3036354138 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f339c105810 0x7f339c107cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.626+0000 7f33a2204700 1 --2- 172.21.15.88:0/3036354138 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f339c069690 0x7f339c1027c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.626+0000 7f33a2204700 1 --2- 172.21.15.88:0/3036354138 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f339c102d00 0x7f339c105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.626+0000 7f33a2204700 1 -- 172.21.15.88:0/3036354138 >> 172.21.15.88:0/3036354138 conn(0x7f339c0fa150 msgr2=0x7f339c0fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:38.609 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.626+0000 7f33a2204700 1 -- 172.21.15.88:0/3036354138 shutdown_connections 2023-07-14T17:44:38.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.626+0000 7f33a2204700 1 -- 172.21.15.88:0/3036354138 wait complete. 2023-07-14T17:44:38.610 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.627+0000 7f33a2204700 1 Processor -- start 2023-07-14T17:44:38.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.627+0000 7f33a2204700 1 -- start start 2023-07-14T17:44:38.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.627+0000 7f33a2204700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f339c069690 0x7f339c1958c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:38.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.627+0000 7f33a2204700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f339c102d00 0x7f339c195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:38.612 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.627+0000 7f33a2204700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f339c105810 0x7f339c19ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:38.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.627+0000 7f33a2204700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339c19d440 con 0x7f339c102d00 2023-07-14T17:44:38.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.627+0000 7f33a2204700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339c19d580 con 0x7f339c069690 2023-07-14T17:44:38.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.627+0000 7f33a2204700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339c19d710 con 0x7f339c105810 2023-07-14T17:44:38.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.627+0000 7f339affd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f339c102d00 0x7f339c195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:38.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.627+0000 7f339affd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f339c102d00 0x7f339c195e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43578/0 (socket says 172.21.15.88:43578) 2023-07-14T17:44:38.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.627+0000 7f339affd700 1 -- 172.21.15.88:0/2810984613 learned_addr learned my addr 172.21.15.88:0/2810984613 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:38.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.627+0000 7f339b7fe700 1 --2- 172.21.15.88:0/2810984613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f339c069690 0x7f339c1958c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:38.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.627+0000 7f339bfff700 1 --2- 172.21.15.88:0/2810984613 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f339c105810 0x7f339c19ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:38.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.627+0000 7f339affd700 1 -- 172.21.15.88:0/2810984613 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f339c105810 msgr2=0x7f339c19ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:38.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.627+0000 7f339affd700 1 --2- 172.21.15.88:0/2810984613 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f339c105810 0x7f339c19ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.627+0000 7f339affd700 1 -- 172.21.15.88:0/2810984613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f339c069690 msgr2=0x7f339c1958c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:38.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.627+0000 7f339affd700 1 --2- 172.21.15.88:0/2810984613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f339c069690 0x7f339c1958c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.627+0000 7f339affd700 1 -- 172.21.15.88:0/2810984613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33900095f0 con 0x7f339c102d00 2023-07-14T17:44:38.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.628+0000 7f339affd700 1 --2- 172.21.15.88:0/2810984613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f339c102d00 0x7f339c195e00 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f338c00b530 tx=0x7f338c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:38.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.628+0000 7f3398ff9700 1 -- 172.21.15.88:0/2810984613 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c011830 con 0x7f339c102d00 2023-07-14T17:44:38.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.628+0000 7f3398ff9700 1 -- 172.21.15.88:0/2810984613 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f338c011e20 con 0x7f339c102d00 2023-07-14T17:44:38.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.628+0000 7f33a2204700 1 -- 172.21.15.88:0/2810984613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f339c19d9f0 con 0x7f339c102d00 2023-07-14T17:44:38.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.628+0000 7f3398ff9700 1 -- 172.21.15.88:0/2810984613 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c010dd0 con 0x7f339c102d00 2023-07-14T17:44:38.616 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.628+0000 7f33a2204700 1 -- 172.21.15.88:0/2810984613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f339c19df80 con 0x7f339c102d00 2023-07-14T17:44:38.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.629+0000 7f33a2204700 1 -- 172.21.15.88:0/2810984613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f339c0fbcc0 con 0x7f339c102d00 2023-07-14T17:44:38.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.630+0000 7f3398ff9700 1 -- 172.21.15.88:0/2810984613 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f338c011990 con 0x7f339c102d00 2023-07-14T17:44:38.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.630+0000 7f3398ff9700 1 --2- 172.21.15.88:0/2810984613 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f33840a11e0 0x7f33840a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:38.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.630+0000 7f339b7fe700 1 --2- 172.21.15.88:0/2810984613 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f33840a11e0 0x7f33840a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:38.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.630+0000 7f3398ff9700 1 -- 172.21.15.88:0/2810984613 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f338c0cc810 con 0x7f339c102d00 2023-07-14T17:44:38.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.631+0000 7f339b7fe700 1 --2- 172.21.15.88:0/2810984613 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f33840a11e0 0x7f33840a3690 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f339000b5c0 tx=0x7f3390005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:38.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.634+0000 7f3398ff9700 1 -- 172.21.15.88:0/2810984613 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f338c095c00 con 0x7f339c102d00 2023-07-14T17:44:38.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.824+0000 7f33a2204700 1 -- 172.21.15.88:0/2810984613 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f339c02caf0 con 0x7f339c102d00 2023-07-14T17:44:38.805 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.824+0000 7f3398ff9700 1 -- 172.21.15.88:0/2810984613 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f338c099220 con 0x7f339c102d00 2023-07-14T17:44:38.806 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-07-14T17:44:38.807 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.826+0000 7f33a2204700 1 -- 172.21.15.88:0/2810984613 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f33840a11e0 msgr2=0x7f33840a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:38.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.826+0000 7f33a2204700 1 --2- 172.21.15.88:0/2810984613 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f33840a11e0 0x7f33840a3690 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f339000b5c0 tx=0x7f3390005fb0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.826+0000 7f33a2204700 1 -- 172.21.15.88:0/2810984613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f339c102d00 msgr2=0x7f339c195e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:38.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.826+0000 7f33a2204700 1 --2- 172.21.15.88:0/2810984613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f339c102d00 0x7f339c195e00 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f338c00b530 tx=0x7f338c00b8f0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.808 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.826+0000 7f33a2204700 1 -- 172.21.15.88:0/2810984613 shutdown_connections 2023-07-14T17:44:38.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.826+0000 7f33a2204700 1 --2- 172.21.15.88:0/2810984613 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f339c105810 0x7f339c19ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.826+0000 7f33a2204700 1 --2- 172.21.15.88:0/2810984613 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f339c069690 0x7f339c1958c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.826+0000 7f33a2204700 1 --2- 172.21.15.88:0/2810984613 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f33840a11e0 0x7f33840a3690 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.809 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.826+0000 7f33a2204700 1 --2- 172.21.15.88:0/2810984613 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f339c102d00 0x7f339c195e00 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.826+0000 7f33a2204700 1 -- 172.21.15.88:0/2810984613 >> 172.21.15.88:0/2810984613 conn(0x7f339c0fa150 msgr2=0x7f339c101060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:38.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.826+0000 7f33a2204700 1 -- 172.21.15.88:0/2810984613 shutdown_connections 2023-07-14T17:44:38.810 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.826+0000 7f33a2204700 1 -- 172.21.15.88:0/2810984613 wait complete. 2023-07-14T17:44:38.819 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T17:44:38.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.963+0000 7f00a97a8700 1 -- 172.21.15.88:0/1222175511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a40fe660 msgr2=0x7f00a4108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:38.944 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.963+0000 7f00a97a8700 1 --2- 172.21.15.88:0/1222175511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a40fe660 0x7f00a4108220 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f009c00b0a0 tx=0x7f009c00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.963+0000 7f00a97a8700 1 -- 172.21.15.88:0/1222175511 shutdown_connections 2023-07-14T17:44:38.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.963+0000 7f00a97a8700 1 --2- 172.21.15.88:0/1222175511 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a40fe660 0x7f00a4108220 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.963+0000 7f00a97a8700 1 --2- 172.21.15.88:0/1222175511 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00a4101240 0x7f00a4101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.945 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.963+0000 7f00a97a8700 1 --2- 172.21.15.88:0/1222175511 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f00a40fdcb0 0x7f00a40fe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.963+0000 7f00a97a8700 1 -- 172.21.15.88:0/1222175511 >> 172.21.15.88:0/1222175511 conn(0x7f00a40f98e0 msgr2=0x7f00a40fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:38.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.963+0000 7f00a97a8700 1 -- 172.21.15.88:0/1222175511 shutdown_connections 2023-07-14T17:44:38.946 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.963+0000 7f00a97a8700 1 -- 172.21.15.88:0/1222175511 wait complete. 2023-07-14T17:44:38.950 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.963+0000 7f00a97a8700 1 Processor -- start 2023-07-14T17:44:38.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.963+0000 7f00a97a8700 1 -- start start 2023-07-14T17:44:38.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a97a8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a40fdcb0 0x7f00a4100940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:38.951 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a97a8700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f00a40fe660 0x7f00a40fef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:38.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a97a8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00a4101240 0x7f00a40ff4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:38.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a97a8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a40ffc60 con 0x7f00a40fdcb0 2023-07-14T17:44:38.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a97a8700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a40ffdd0 con 0x7f00a4101240 2023-07-14T17:44:38.952 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a97a8700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a419f920 con 0x7f00a40fe660 2023-07-14T17:44:38.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a2ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a40fdcb0 0x7f00a4100940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:38.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a2ffd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a40fdcb0 0x7f00a4100940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43590/0 (socket says 172.21.15.88:43590) 2023-07-14T17:44:38.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a2ffd700 1 -- 172.21.15.88:0/2126025500 learned_addr learned my addr 172.21.15.88:0/2126025500 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:38.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a27fc700 1 --2- 172.21.15.88:0/2126025500 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f00a40fe660 0x7f00a40fef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:38.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a37fe700 1 --2- 172.21.15.88:0/2126025500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00a4101240 0x7f00a40ff4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:38.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a2ffd700 1 -- 172.21.15.88:0/2126025500 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f00a40fe660 msgr2=0x7f00a40fef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:38.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a2ffd700 1 --2- 172.21.15.88:0/2126025500 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f00a40fe660 0x7f00a40fef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a2ffd700 1 -- 172.21.15.88:0/2126025500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00a4101240 msgr2=0x7f00a40ff4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:38.954 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a2ffd700 1 --2- 172.21.15.88:0/2126025500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00a4101240 0x7f00a40ff4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:38.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a2ffd700 1 -- 172.21.15.88:0/2126025500 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c009d00 con 0x7f00a40fdcb0 2023-07-14T17:44:38.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a27fc700 1 --2- 172.21.15.88:0/2126025500 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f00a40fe660 0x7f00a40fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:38.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a2ffd700 1 --2- 172.21.15.88:0/2126025500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a40fdcb0 0x7f00a4100940 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f009800e970 tx=0x7f009800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:38.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f008bfff700 1 -- 172.21.15.88:0/2126025500 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0098009790 con 0x7f00a40fdcb0 2023-07-14T17:44:38.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a97a8700 1 -- 172.21.15.88:0/2126025500 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a419fc00 con 0x7f00a40fdcb0 2023-07-14T17:44:38.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f008bfff700 1 -- 172.21.15.88:0/2126025500 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f00980044b0 con 0x7f00a40fdcb0 2023-07-14T17:44:38.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f008bfff700 1 -- 172.21.15.88:0/2126025500 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0098003dd0 con 0x7f00a40fdcb0 2023-07-14T17:44:38.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.964+0000 7f00a97a8700 1 -- 172.21.15.88:0/2126025500 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a41a0110 con 0x7f00a40fdcb0 2023-07-14T17:44:38.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.965+0000 7f00a97a8700 1 -- 172.21.15.88:0/2126025500 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a4106c70 con 0x7f00a40fdcb0 2023-07-14T17:44:38.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.966+0000 7f008bfff700 1 -- 172.21.15.88:0/2126025500 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f0098003f30 con 0x7f00a40fdcb0 2023-07-14T17:44:38.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.967+0000 7f008bfff700 1 --2- 172.21.15.88:0/2126025500 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f008c0a11e0 0x7f008c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:38.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.967+0000 7f008bfff700 1 -- 172.21.15.88:0/2126025500 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0098014070 con 0x7f00a40fdcb0 2023-07-14T17:44:38.958 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.967+0000 7f00a27fc700 1 --2- 172.21.15.88:0/2126025500 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f008c0a11e0 0x7f008c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:38.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.967+0000 7f00a27fc700 1 --2- 172.21.15.88:0/2126025500 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f008c0a11e0 0x7f008c0a3690 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f00a4068900 tx=0x7f009400a680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:38.959 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:38.971+0000 7f008bfff700 1 -- 172.21.15.88:0/2126025500 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f00980963d0 con 0x7f00a40fdcb0 2023-07-14T17:44:39.141 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.160+0000 7f00a97a8700 1 -- 172.21.15.88:0/2126025500 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f00a402cb90 con 0x7f00a40fdcb0 2023-07-14T17:44:39.142 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.160+0000 7f008bfff700 1 -- 172.21.15.88:0/2126025500 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f00980999f0 con 0x7f00a40fdcb0 2023-07-14T17:44:39.142 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 64 2023-07-14T17:44:39.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.162+0000 7f00a97a8700 1 -- 172.21.15.88:0/2126025500 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f008c0a11e0 msgr2=0x7f008c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:39.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.162+0000 7f00a97a8700 1 --2- 172.21.15.88:0/2126025500 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f008c0a11e0 0x7f008c0a3690 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f00a4068900 tx=0x7f009400a680 comp rx=0 tx=0).stop 2023-07-14T17:44:39.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.162+0000 7f00a97a8700 1 -- 172.21.15.88:0/2126025500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a40fdcb0 msgr2=0x7f00a4100940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:39.144 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.162+0000 7f00a97a8700 1 --2- 172.21.15.88:0/2126025500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a40fdcb0 0x7f00a4100940 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f009800e970 tx=0x7f009800ed30 comp rx=0 tx=0).stop 2023-07-14T17:44:39.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.162+0000 7f00a97a8700 1 -- 172.21.15.88:0/2126025500 shutdown_connections 2023-07-14T17:44:39.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.162+0000 7f00a97a8700 1 --2- 172.21.15.88:0/2126025500 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f00a4101240 0x7f00a40ff4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.162+0000 7f00a97a8700 1 --2- 172.21.15.88:0/2126025500 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f008c0a11e0 0x7f008c0a3690 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.162+0000 7f00a97a8700 1 --2- 172.21.15.88:0/2126025500 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f00a40fdcb0 0x7f00a4100940 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.145 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.162+0000 7f00a97a8700 1 --2- 172.21.15.88:0/2126025500 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f00a40fe660 0x7f00a40fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.162+0000 7f00a97a8700 1 -- 172.21.15.88:0/2126025500 >> 172.21.15.88:0/2126025500 conn(0x7f00a40f98e0 msgr2=0x7f00a40fbac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:39.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.163+0000 7f00a97a8700 1 -- 172.21.15.88:0/2126025500 shutdown_connections 2023-07-14T17:44:39.146 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.163+0000 7f00a97a8700 1 -- 172.21.15.88:0/2126025500 wait complete. 2023-07-14T17:44:39.155 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:39 smithi088 ceph-mon[127465]: pgmap v315: 129 pgs: 129 active+clean; 644 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1023 B/s wr, 10 op/s 2023-07-14T17:44:39.155 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:39 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3310910981' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:39.155 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:39 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1325378940' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:44:39.156 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:39 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2810984613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:44:39.157 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T17:44:39.281 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.300+0000 7fc65c5ed700 1 -- 172.21.15.88:0/3428067598 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc654105930 msgr2=0x7fc654107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:39.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.300+0000 7fc65c5ed700 1 --2- 172.21.15.88:0/3428067598 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc654105930 0x7fc654107d30 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00b0a0 tx=0x7fc64c00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.300+0000 7fc65c5ed700 1 -- 172.21.15.88:0/3428067598 shutdown_connections 2023-07-14T17:44:39.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.300+0000 7fc65c5ed700 1 --2- 172.21.15.88:0/3428067598 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc654105930 0x7fc654107d30 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.300+0000 7fc65c5ed700 1 --2- 172.21.15.88:0/3428067598 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc6541006f0 0x7fc654102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.282 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.300+0000 7fc65c5ed700 1 --2- 172.21.15.88:0/3428067598 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc654103010 0x7fc6541053f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.300+0000 7fc65c5ed700 1 -- 172.21.15.88:0/3428067598 >> 172.21.15.88:0/3428067598 conn(0x7fc6540fa130 msgr2=0x7fc6540fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:39.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.300+0000 7fc65c5ed700 1 -- 172.21.15.88:0/3428067598 shutdown_connections 2023-07-14T17:44:39.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.300+0000 7fc65c5ed700 1 -- 172.21.15.88:0/3428067598 wait complete. 2023-07-14T17:44:39.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.300+0000 7fc65c5ed700 1 Processor -- start 2023-07-14T17:44:39.283 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.300+0000 7fc65c5ed700 1 -- start start 2023-07-14T17:44:39.285 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.300+0000 7fc65c5ed700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6541006f0 0x7fc654195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:39.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.300+0000 7fc65c5ed700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc654103010 0x7fc654195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:39.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.300+0000 7fc65c5ed700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc654105930 0x7fc65419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:39.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.300+0000 7fc65c5ed700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65419d4b0 con 0x7fc6541006f0 2023-07-14T17:44:39.286 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.300+0000 7fc65c5ed700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65419d5f0 con 0x7fc654103010 2023-07-14T17:44:39.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.300+0000 7fc65c5ed700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc65419d780 con 0x7fc654105930 2023-07-14T17:44:39.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.301+0000 7fc65a389700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6541006f0 0x7fc654195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:39.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.301+0000 7fc65a389700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6541006f0 0x7fc654195930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43604/0 (socket says 172.21.15.88:43604) 2023-07-14T17:44:39.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.301+0000 7fc65a389700 1 -- 172.21.15.88:0/1200580404 learned_addr learned my addr 172.21.15.88:0/1200580404 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:39.287 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.301+0000 7fc659b88700 1 --2- 172.21.15.88:0/1200580404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc654103010 0x7fc654195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:39.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.301+0000 7fc65a389700 1 -- 172.21.15.88:0/1200580404 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc654105930 msgr2=0x7fc65419cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:39.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.301+0000 7fc65ab8a700 1 --2- 172.21.15.88:0/1200580404 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc654105930 0x7fc65419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:39.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.301+0000 7fc65a389700 1 --2- 172.21.15.88:0/1200580404 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc654105930 0x7fc65419cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.301+0000 7fc65a389700 1 -- 172.21.15.88:0/1200580404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc654103010 msgr2=0x7fc654195e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:39.288 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.301+0000 7fc65a389700 1 --2- 172.21.15.88:0/1200580404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc654103010 0x7fc654195e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.301+0000 7fc65a389700 1 -- 172.21.15.88:0/1200580404 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c009d00 con 0x7fc6541006f0 2023-07-14T17:44:39.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.301+0000 7fc65a389700 1 --2- 172.21.15.88:0/1200580404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6541006f0 0x7fc654195930 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc64400c920 tx=0x7fc64400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:39.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.301+0000 7fc64b7fe700 1 -- 172.21.15.88:0/1200580404 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644007b00 con 0x7fc6541006f0 2023-07-14T17:44:39.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.301+0000 7fc64b7fe700 1 -- 172.21.15.88:0/1200580404 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc64400ce70 con 0x7fc6541006f0 2023-07-14T17:44:39.289 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.301+0000 7fc65c5ed700 1 -- 172.21.15.88:0/1200580404 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65419da60 con 0x7fc6541006f0 2023-07-14T17:44:39.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.301+0000 7fc64b7fe700 1 -- 172.21.15.88:0/1200580404 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644005440 con 0x7fc6541006f0 2023-07-14T17:44:39.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.301+0000 7fc65c5ed700 1 -- 172.21.15.88:0/1200580404 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65419dfc0 con 0x7fc6541006f0 2023-07-14T17:44:39.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.302+0000 7fc65c5ed700 1 -- 172.21.15.88:0/1200580404 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6540fbca0 con 0x7fc6541006f0 2023-07-14T17:44:39.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.303+0000 7fc64b7fe700 1 -- 172.21.15.88:0/1200580404 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc6440055a0 con 0x7fc6541006f0 2023-07-14T17:44:39.290 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.304+0000 7fc64b7fe700 1 --2- 172.21.15.88:0/1200580404 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc6400a1260 0x7fc6400a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:39.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.304+0000 7fc659b88700 1 --2- 172.21.15.88:0/1200580404 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc6400a1260 0x7fc6400a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:39.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.304+0000 7fc64b7fe700 1 -- 172.21.15.88:0/1200580404 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc6440ccda0 con 0x7fc6541006f0 2023-07-14T17:44:39.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.304+0000 7fc659b88700 1 --2- 172.21.15.88:0/1200580404 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc6400a1260 0x7fc6400a3710 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc650005950 tx=0x7fc65000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:39.291 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.307+0000 7fc64b7fe700 1 -- 172.21.15.88:0/1200580404 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc6440960a0 con 0x7fc6541006f0 2023-07-14T17:44:39.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:39 smithi178 ceph-mon[129368]: pgmap v315: 129 pgs: 129 active+clean; 644 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1023 B/s wr, 10 op/s 2023-07-14T17:44:39.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:39 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3310910981' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:39.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:39 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1325378940' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:44:39.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:39 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2810984613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:44:39.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:39 smithi107 ceph-mon[128999]: pgmap v315: 129 pgs: 129 active+clean; 644 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1023 B/s wr, 10 op/s 2023-07-14T17:44:39.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:39 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3310910981' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:39.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:39 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1325378940' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T17:44:39.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:39 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2810984613' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T17:44:39.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.495+0000 7fc65c5ed700 1 -- 172.21.15.88:0/1200580404 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc65402cb50 con 0x7fc6541006f0 2023-07-14T17:44:39.476 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.495+0000 7fc64b7fe700 1 -- 172.21.15.88:0/1200580404 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v88) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc64401d070 con 0x7fc6541006f0 2023-07-14T17:44:39.477 INFO:teuthology.orchestra.run.smithi088.stdout:pg_num: 32 2023-07-14T17:44:39.478 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.497+0000 7fc65c5ed700 1 -- 172.21.15.88:0/1200580404 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc6400a1260 msgr2=0x7fc6400a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:39.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.497+0000 7fc65c5ed700 1 --2- 172.21.15.88:0/1200580404 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc6400a1260 0x7fc6400a3710 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc650005950 tx=0x7fc65000a300 comp rx=0 tx=0).stop 2023-07-14T17:44:39.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.497+0000 7fc65c5ed700 1 -- 172.21.15.88:0/1200580404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6541006f0 msgr2=0x7fc654195930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:39.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.497+0000 7fc65c5ed700 1 --2- 172.21.15.88:0/1200580404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6541006f0 0x7fc654195930 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc64400c920 tx=0x7fc64400cc30 comp rx=0 tx=0).stop 2023-07-14T17:44:39.479 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.497+0000 7fc65c5ed700 1 -- 172.21.15.88:0/1200580404 shutdown_connections 2023-07-14T17:44:39.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.497+0000 7fc65c5ed700 1 --2- 172.21.15.88:0/1200580404 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc654105930 0x7fc65419cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.497+0000 7fc65c5ed700 1 --2- 172.21.15.88:0/1200580404 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc6400a1260 0x7fc6400a3710 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.497+0000 7fc65c5ed700 1 --2- 172.21.15.88:0/1200580404 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc6541006f0 0x7fc654195930 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.480 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.497+0000 7fc65c5ed700 1 --2- 172.21.15.88:0/1200580404 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc654103010 0x7fc654195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.497+0000 7fc65c5ed700 1 -- 172.21.15.88:0/1200580404 >> 172.21.15.88:0/1200580404 conn(0x7fc6540fa130 msgr2=0x7fc654101350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:39.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.497+0000 7fc65c5ed700 1 -- 172.21.15.88:0/1200580404 shutdown_connections 2023-07-14T17:44:39.481 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.497+0000 7fc65c5ed700 1 -- 172.21.15.88:0/1200580404 wait complete. 2023-07-14T17:44:39.491 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T17:44:39.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.631+0000 7fec549c3700 1 -- 172.21.15.88:0/4038822920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec500feb10 msgr2=0x7fec500fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:39.613 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.631+0000 7fec549c3700 1 --2- 172.21.15.88:0/4038822920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec500feb10 0x7fec500fef20 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fec40009990 tx=0x7fec40009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.632+0000 7fec549c3700 1 -- 172.21.15.88:0/4038822920 shutdown_connections 2023-07-14T17:44:39.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.632+0000 7fec549c3700 1 --2- 172.21.15.88:0/4038822920 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec50100690 0x7fec50106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.632+0000 7fec549c3700 1 --2- 172.21.15.88:0/4038822920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec500feb10 0x7fec500fef20 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.614 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.632+0000 7fec549c3700 1 --2- 172.21.15.88:0/4038822920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec500ffd00 0x7fec50100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.632+0000 7fec549c3700 1 -- 172.21.15.88:0/4038822920 >> 172.21.15.88:0/4038822920 conn(0x7fec500fa130 msgr2=0x7fec500fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:39.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.632+0000 7fec549c3700 1 -- 172.21.15.88:0/4038822920 shutdown_connections 2023-07-14T17:44:39.615 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.632+0000 7fec549c3700 1 -- 172.21.15.88:0/4038822920 wait complete. 2023-07-14T17:44:39.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.632+0000 7fec549c3700 1 Processor -- start 2023-07-14T17:44:39.617 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.632+0000 7fec549c3700 1 -- start start 2023-07-14T17:44:39.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec549c3700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec500feb10 0x7fec50069360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:39.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec549c3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec500ffd00 0x7fec5019a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:39.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec549c3700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec50100690 0x7fec5019cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:39.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec549c3700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5019d420 con 0x7fec500ffd00 2023-07-14T17:44:39.618 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec549c3700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5019d590 con 0x7fec50100690 2023-07-14T17:44:39.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec549c3700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5019d750 con 0x7fec500feb10 2023-07-14T17:44:39.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec4dd9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec500ffd00 0x7fec5019a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:39.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec4dd9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec500ffd00 0x7fec5019a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43620/0 (socket says 172.21.15.88:43620) 2023-07-14T17:44:39.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec4dd9b700 1 -- 172.21.15.88:0/3052014662 learned_addr learned my addr 172.21.15.88:0/3052014662 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:39.619 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec4e59c700 1 --2- 172.21.15.88:0/3052014662 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec500feb10 0x7fec50069360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:39.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec4ed9d700 1 --2- 172.21.15.88:0/3052014662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec50100690 0x7fec5019cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:39.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec4dd9b700 1 -- 172.21.15.88:0/3052014662 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec500feb10 msgr2=0x7fec50069360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:39.620 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec4dd9b700 1 --2- 172.21.15.88:0/3052014662 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec500feb10 0x7fec50069360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec4dd9b700 1 -- 172.21.15.88:0/3052014662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec50100690 msgr2=0x7fec5019cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:39.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec4dd9b700 1 --2- 172.21.15.88:0/3052014662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec50100690 0x7fec5019cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec4dd9b700 1 -- 172.21.15.88:0/3052014662 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec400095f0 con 0x7fec500ffd00 2023-07-14T17:44:39.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec4e59c700 1 --2- 172.21.15.88:0/3052014662 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec500feb10 0x7fec50069360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:39.621 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec4dd9b700 1 --2- 172.21.15.88:0/3052014662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec500ffd00 0x7fec5019a600 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fec4400d950 tx=0x7fec4400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:39.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec3f7fe700 1 -- 172.21.15.88:0/3052014662 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec44009b40 con 0x7fec500ffd00 2023-07-14T17:44:39.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec3f7fe700 1 -- 172.21.15.88:0/3052014662 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fec44009ca0 con 0x7fec500ffd00 2023-07-14T17:44:39.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec549c3700 1 -- 172.21.15.88:0/3052014662 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5019da30 con 0x7fec500ffd00 2023-07-14T17:44:39.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec3f7fe700 1 -- 172.21.15.88:0/3052014662 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4400f620 con 0x7fec500ffd00 2023-07-14T17:44:39.622 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.633+0000 7fec549c3700 1 -- 172.21.15.88:0/3052014662 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec5019df40 con 0x7fec500ffd00 2023-07-14T17:44:39.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.634+0000 7fec549c3700 1 -- 172.21.15.88:0/3052014662 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec5010a190 con 0x7fec500ffd00 2023-07-14T17:44:39.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.635+0000 7fec3f7fe700 1 -- 172.21.15.88:0/3052014662 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fec44010460 con 0x7fec500ffd00 2023-07-14T17:44:39.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.636+0000 7fec3f7fe700 1 --2- 172.21.15.88:0/3052014662 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fec380a11e0 0x7fec380a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:39.623 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.636+0000 7fec3f7fe700 1 -- 172.21.15.88:0/3052014662 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fec440cd120 con 0x7fec500ffd00 2023-07-14T17:44:39.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.636+0000 7fec4e59c700 1 --2- 172.21.15.88:0/3052014662 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fec380a11e0 0x7fec380a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:39.624 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.636+0000 7fec4e59c700 1 --2- 172.21.15.88:0/3052014662 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fec380a11e0 0x7fec380a3690 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fec4000c010 tx=0x7fec4000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:39.625 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.639+0000 7fec3f7fe700 1 -- 172.21.15.88:0/3052014662 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fec44096400 con 0x7fec500ffd00 2023-07-14T17:44:39.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.841+0000 7fec549c3700 1 -- 172.21.15.88:0/3052014662 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fec50105ac0 con 0x7fec500ffd00 2023-07-14T17:44:39.823 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.841+0000 7fec3f7fe700 1 -- 172.21.15.88:0/3052014662 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v88) v1 ==== 81+0+591 (secure 0 0 0) 0x7fec44016260 con 0x7fec500ffd00 2023-07-14T17:44:39.824 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/1441439704 2023-07-15T17:36:43.799741+0000 2023-07-14T17:44:39.824 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/3897985004 2023-07-15T17:36:43.799741+0000 2023-07-14T17:44:39.825 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6801/2424184187 2023-07-15T17:36:43.799741+0000 2023-07-14T17:44:39.825 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6801/1369181448 2023-07-15T17:36:21.153428+0000 2023-07-14T17:44:39.826 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6800/1369181448 2023-07-15T17:36:21.153428+0000 2023-07-14T17:44:39.826 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/1790623516 2023-07-15T17:36:21.153428+0000 2023-07-14T17:44:39.826 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/3232411191 2023-07-15T17:36:21.153428+0000 2023-07-14T17:44:39.827 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/3329866608 2023-07-15T17:36:21.153428+0000 2023-07-14T17:44:39.827 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:0/954461155 2023-07-15T17:36:43.799741+0000 2023-07-14T17:44:39.827 INFO:teuthology.orchestra.run.smithi088.stdout:172.21.15.88:6800/2424184187 2023-07-15T17:36:43.799741+0000 2023-07-14T17:44:39.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.844+0000 7fec549c3700 1 -- 172.21.15.88:0/3052014662 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fec380a11e0 msgr2=0x7fec380a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:39.829 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.844+0000 7fec549c3700 1 --2- 172.21.15.88:0/3052014662 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fec380a11e0 0x7fec380a3690 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fec4000c010 tx=0x7fec4000b540 comp rx=0 tx=0).stop 2023-07-14T17:44:39.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.844+0000 7fec549c3700 1 -- 172.21.15.88:0/3052014662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec500ffd00 msgr2=0x7fec5019a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:39.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.844+0000 7fec549c3700 1 --2- 172.21.15.88:0/3052014662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec500ffd00 0x7fec5019a600 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fec4400d950 tx=0x7fec4400dd10 comp rx=0 tx=0).stop 2023-07-14T17:44:39.830 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.844+0000 7fec549c3700 1 -- 172.21.15.88:0/3052014662 shutdown_connections 2023-07-14T17:44:39.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.844+0000 7fec549c3700 1 --2- 172.21.15.88:0/3052014662 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fec50100690 0x7fec5019cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.831 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.844+0000 7fec549c3700 1 --2- 172.21.15.88:0/3052014662 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fec500feb10 0x7fec50069360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.844+0000 7fec549c3700 1 --2- 172.21.15.88:0/3052014662 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fec380a11e0 0x7fec380a3690 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.832 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.844+0000 7fec549c3700 1 --2- 172.21.15.88:0/3052014662 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec500ffd00 0x7fec5019a600 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:39.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.844+0000 7fec549c3700 1 -- 172.21.15.88:0/3052014662 >> 172.21.15.88:0/3052014662 conn(0x7fec500fa130 msgr2=0x7fec500fc420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:39.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.844+0000 7fec549c3700 1 -- 172.21.15.88:0/3052014662 shutdown_connections 2023-07-14T17:44:39.833 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:39.844+0000 7fec549c3700 1 -- 172.21.15.88:0/3052014662 wait complete. 2023-07-14T17:44:39.834 INFO:teuthology.orchestra.run.smithi088.stderr:listed 10 entries 2023-07-14T17:44:39.837 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi088.front.sepia.ceph.com... 2023-07-14T17:44:39.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:39.839 DEBUG:teuthology.orchestra.run.smithi088:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-07-14T17:44:39.878 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi088.stderr:ceph-fuse[178582]: fuse finished with error 0 and tester_r 0 2023-07-14T17:44:39.880 INFO:teuthology.orchestra.run:waiting for 300 2023-07-14T17:44:40.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:40 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2126025500' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:44:40.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:40 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/1200580404' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:44:40.192 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:40 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3052014662' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T17:44:40.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:40 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2126025500' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:44:40.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:40 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/1200580404' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:44:40.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:40 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3052014662' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T17:44:40.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:40 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2126025500' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T17:44:40.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:40 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/1200580404' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T17:44:40.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:40 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3052014662' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T17:44:41.191 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:41 smithi088 ceph-mon[127465]: pgmap v316: 129 pgs: 129 active+clean; 645 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2023-07-14T17:44:41.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:41 smithi178 ceph-mon[129368]: pgmap v316: 129 pgs: 129 active+clean; 645 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2023-07-14T17:44:41.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:41 smithi107 ceph-mon[128999]: pgmap v316: 129 pgs: 129 active+clean; 645 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2023-07-14T17:44:43.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:43 smithi178 ceph-mon[129368]: pgmap v317: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.0 KiB/s wr, 14 op/s 2023-07-14T17:44:43.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:43 smithi088 ceph-mon[127465]: pgmap v317: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.0 KiB/s wr, 14 op/s 2023-07-14T17:44:43.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:43 smithi107 ceph-mon[128999]: pgmap v317: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.0 KiB/s wr, 14 op/s 2023-07-14T17:44:44.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2023-07-14T17:44:44.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:44 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 10]}]: dispatch 2023-07-14T17:44:44.440 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2023-07-14T17:44:44.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:44 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 10]}]: dispatch 2023-07-14T17:44:44.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2023-07-14T17:44:44.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:44 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 10]}]: dispatch 2023-07-14T17:44:45.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:45 smithi178 ceph-mon[129368]: pgmap v318: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2023-07-14T17:44:45.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:45 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2023-07-14T17:44:45.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:45 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 10]}]': finished 2023-07-14T17:44:45.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:45 smithi178 ceph-mon[129368]: osdmap e89: 12 total, 12 up, 12 in 2023-07-14T17:44:45.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:45 smithi088 ceph-mon[127465]: pgmap v318: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2023-07-14T17:44:45.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:45 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2023-07-14T17:44:45.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:45 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 10]}]': finished 2023-07-14T17:44:45.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:45 smithi088 ceph-mon[127465]: osdmap e89: 12 total, 12 up, 12 in 2023-07-14T17:44:45.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:45 smithi107 ceph-mon[128999]: pgmap v318: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 2.1 KiB/s wr, 12 op/s 2023-07-14T17:44:45.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:45 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2023-07-14T17:44:45.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:45 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [5, 10]}]': finished 2023-07-14T17:44:45.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:45 smithi107 ceph-mon[128999]: osdmap e89: 12 total, 12 up, 12 in 2023-07-14T17:44:45.982 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi088.front.sepia.ceph.com 2023-07-14T17:44:45.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:45.983 DEBUG:teuthology.orchestra.run.smithi088:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-07-14T17:44:46.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:46.006 DEBUG:teuthology.orchestra.run.smithi088:> ip netns list 2023-07-14T17:44:46.066 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-07-14T17:44:46.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:46.067 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-14T17:44:46.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T17:44:46.152 DEBUG:teuthology.orchestra.run.smithi088:> sudo ip link delete ceph-brx 2023-07-14T17:44:46.268 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-14T17:44:46.280 INFO:tasks.cephadm:Teardown begin 2023-07-14T17:44:46.280 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-07-14T17:44:46.281 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:44:46.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.424+0000 7f6d8fe3e700 1 -- 172.21.15.88:0/1654421001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d880ffd20 msgr2=0x7f6d88100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:46.407 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.424+0000 7f6d8fe3e700 1 --2- 172.21.15.88:0/1654421001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d880ffd20 0x7f6d88100170 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009990 tx=0x7f6d78009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:46.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.424+0000 7f6d8fe3e700 1 -- 172.21.15.88:0/1654421001 shutdown_connections 2023-07-14T17:44:46.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.424+0000 7f6d8fe3e700 1 --2- 172.21.15.88:0/1654421001 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d881006b0 0x7f6d88106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:46.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.424+0000 7f6d8fe3e700 1 --2- 172.21.15.88:0/1654421001 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d880feb30 0x7f6d880fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:46.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.424+0000 7f6d8fe3e700 1 --2- 172.21.15.88:0/1654421001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d880ffd20 0x7f6d88100170 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:46.408 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.424+0000 7f6d8fe3e700 1 -- 172.21.15.88:0/1654421001 >> 172.21.15.88:0/1654421001 conn(0x7f6d880fa150 msgr2=0x7f6d880fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:46.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.424+0000 7f6d8fe3e700 1 -- 172.21.15.88:0/1654421001 shutdown_connections 2023-07-14T17:44:46.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.424+0000 7f6d8fe3e700 1 -- 172.21.15.88:0/1654421001 wait complete. 2023-07-14T17:44:46.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8fe3e700 1 Processor -- start 2023-07-14T17:44:46.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8fe3e700 1 -- start start 2023-07-14T17:44:46.409 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8fe3e700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d880feb30 0x7f6d88197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:46.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8fe3e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d880ffd20 0x7f6d88197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:46.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8fe3e700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d881006b0 0x7f6d8819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:46.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8fe3e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8819f5a0 con 0x7f6d880ffd20 2023-07-14T17:44:46.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8fe3e700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8819f6e0 con 0x7f6d881006b0 2023-07-14T17:44:46.410 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8fe3e700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8819f870 con 0x7f6d880feb30 2023-07-14T17:44:46.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8d3d9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d880ffd20 0x7f6d88197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:46.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8d3d9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d880ffd20 0x7f6d88197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43632/0 (socket says 172.21.15.88:43632) 2023-07-14T17:44:46.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8d3d9700 1 -- 172.21.15.88:0/3781083763 learned_addr learned my addr 172.21.15.88:0/3781083763 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:46.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8e3db700 1 --2- 172.21.15.88:0/3781083763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d881006b0 0x7f6d8819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:46.411 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8dbda700 1 --2- 172.21.15.88:0/3781083763 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d880feb30 0x7f6d88197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:46.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8d3d9700 1 -- 172.21.15.88:0/3781083763 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d880feb30 msgr2=0x7f6d88197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:46.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8d3d9700 1 --2- 172.21.15.88:0/3781083763 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d880feb30 0x7f6d88197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:46.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8d3d9700 1 -- 172.21.15.88:0/3781083763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d881006b0 msgr2=0x7f6d8819efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:46.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8d3d9700 1 --2- 172.21.15.88:0/3781083763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d881006b0 0x7f6d8819efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:46.412 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8d3d9700 1 -- 172.21.15.88:0/3781083763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d780095f0 con 0x7f6d880ffd20 2023-07-14T17:44:46.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8e3db700 1 --2- 172.21.15.88:0/3781083763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d881006b0 0x7f6d8819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T17:44:46.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.425+0000 7f6d8dbda700 1 --2- 172.21.15.88:0/3781083763 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d880feb30 0x7f6d88197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:46.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.426+0000 7f6d8d3d9700 1 --2- 172.21.15.88:0/3781083763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d880ffd20 0x7f6d88197f60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6d780092e0 tx=0x7f6d78003cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:46.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.426+0000 7f6d7effd700 1 -- 172.21.15.88:0/3781083763 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7801d070 con 0x7f6d880ffd20 2023-07-14T17:44:46.413 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.426+0000 7f6d7effd700 1 -- 172.21.15.88:0/3781083763 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6d7800be00 con 0x7f6d880ffd20 2023-07-14T17:44:46.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.426+0000 7f6d8fe3e700 1 -- 172.21.15.88:0/3781083763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8819faf0 con 0x7f6d880ffd20 2023-07-14T17:44:46.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.426+0000 7f6d7effd700 1 -- 172.21.15.88:0/3781083763 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d78004bb0 con 0x7f6d880ffd20 2023-07-14T17:44:46.414 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.426+0000 7f6d8fe3e700 1 -- 172.21.15.88:0/3781083763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8819feb0 con 0x7f6d880ffd20 2023-07-14T17:44:46.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:46 smithi178 ceph-mon[129368]: osdmap e90: 12 total, 12 up, 12 in 2023-07-14T17:44:46.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.427+0000 7f6d8fe3e700 1 -- 172.21.15.88:0/3781083763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d8810a220 con 0x7f6d880ffd20 2023-07-14T17:44:46.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.428+0000 7f6d7effd700 1 -- 172.21.15.88:0/3781083763 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6d78004d10 con 0x7f6d880ffd20 2023-07-14T17:44:46.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.428+0000 7f6d7effd700 1 --2- 172.21.15.88:0/3781083763 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6d740a1220 0x7f6d740a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:46.416 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.429+0000 7f6d8dbda700 1 --2- 172.21.15.88:0/3781083763 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6d740a1220 0x7f6d740a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:46.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.429+0000 7f6d7effd700 1 -- 172.21.15.88:0/3781083763 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f6d780ce5f0 con 0x7f6d880ffd20 2023-07-14T17:44:46.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.429+0000 7f6d8dbda700 1 --2- 172.21.15.88:0/3781083763 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6d740a1220 0x7f6d740a36d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6d880ffb80 tx=0x7f6d84009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:46.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.432+0000 7f6d7effd700 1 -- 172.21.15.88:0/3781083763 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6d780978c0 con 0x7f6d880ffd20 2023-07-14T17:44:46.440 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:46 smithi088 ceph-mon[127465]: osdmap e90: 12 total, 12 up, 12 in 2023-07-14T17:44:46.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:46 smithi107 ceph-mon[128999]: osdmap e90: 12 total, 12 up, 12 in 2023-07-14T17:44:46.650 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.669+0000 7f6d8fe3e700 1 -- 172.21.15.88:0/3781083763 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d881a0520 con 0x7f6d880ffd20 2023-07-14T17:44:46.651 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.670+0000 7f6d7effd700 1 -- 172.21.15.88:0/3781083763 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10885 (secure 0 0 0) 0x7f6d7809aee0 con 0x7f6d880ffd20 2023-07-14T17:44:46.651 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:46.652 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T17:42:42.314442+0000","modified":"2023-07-14T17:44:35.037715+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24473,"mds_2":24469,"mds_3":24454,"mds_4":24449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6839/3046592155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3046592155},{"type":"v1","addr":"172.21.15.88:6839","nonce":3046592155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6841/2509502918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6840","nonce":2509502918},{"type":"v1","addr":"172.21.15.88:6841","nonce":2509502918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6835/2332025068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":2332025068},{"type":"v1","addr":"172.21.15.178:6835","nonce":2332025068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24449":{"gid":24449,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.107:6835/3062637672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3062637672},{"type":"v1","addr":"172.21.15.107:6835","nonce":3062637672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.178:6837/3950404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3950404084},{"type":"v1","addr":"172.21.15.178:6837","nonce":3950404084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.178:6839/2953415077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2953415077},{"type":"v1","addr":"172.21.15.178:6839","nonce":2953415077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/1152608615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1152608615},{"type":"v1","addr":"172.21.15.107:6837","nonce":1152608615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6839/31646548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":31646548},{"type":"v1","addr":"172.21.15.107:6839","nonce":31646548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T17:44:46.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.672+0000 7f6d8fe3e700 1 -- 172.21.15.88:0/3781083763 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6d740a1220 msgr2=0x7f6d740a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:46.653 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.672+0000 7f6d8fe3e700 1 --2- 172.21.15.88:0/3781083763 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6d740a1220 0x7f6d740a36d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6d880ffb80 tx=0x7f6d84009260 comp rx=0 tx=0).stop 2023-07-14T17:44:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.672+0000 7f6d8fe3e700 1 -- 172.21.15.88:0/3781083763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d880ffd20 msgr2=0x7f6d88197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.672+0000 7f6d8fe3e700 1 --2- 172.21.15.88:0/3781083763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d880ffd20 0x7f6d88197f60 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f6d780092e0 tx=0x7f6d78003cd0 comp rx=0 tx=0).stop 2023-07-14T17:44:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.672+0000 7f6d8fe3e700 1 -- 172.21.15.88:0/3781083763 shutdown_connections 2023-07-14T17:44:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.672+0000 7f6d8fe3e700 1 --2- 172.21.15.88:0/3781083763 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f6d881006b0 0x7f6d8819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:46.654 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.672+0000 7f6d8fe3e700 1 --2- 172.21.15.88:0/3781083763 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f6d880feb30 0x7f6d88197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:46.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.672+0000 7f6d8fe3e700 1 --2- 172.21.15.88:0/3781083763 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f6d740a1220 0x7f6d740a36d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:46.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.672+0000 7f6d8fe3e700 1 --2- 172.21.15.88:0/3781083763 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6d880ffd20 0x7f6d88197f60 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:46.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.672+0000 7f6d8fe3e700 1 -- 172.21.15.88:0/3781083763 >> 172.21.15.88:0/3781083763 conn(0x7f6d880fa150 msgr2=0x7f6d88068a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:46.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.672+0000 7f6d8fe3e700 1 -- 172.21.15.88:0/3781083763 shutdown_connections 2023-07-14T17:44:46.655 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.672+0000 7f6d8fe3e700 1 -- 172.21.15.88:0/3781083763 wait complete. 2023-07-14T17:44:46.656 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 28 2023-07-14T17:44:46.665 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T17:44:46.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.806+0000 7f081abc0700 1 -- 172.21.15.88:0/3619738522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08141006a0 msgr2=0x7f0814106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:46.788 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.806+0000 7f081abc0700 1 --2- 172.21.15.88:0/3619738522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08141006a0 0x7f0814106f60 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f080c00b0a0 tx=0x7f080c00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:46.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.807+0000 7f081abc0700 1 -- 172.21.15.88:0/3619738522 shutdown_connections 2023-07-14T17:44:46.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.807+0000 7f081abc0700 1 --2- 172.21.15.88:0/3619738522 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08141006a0 0x7f0814106f60 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:46.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.807+0000 7f081abc0700 1 --2- 172.21.15.88:0/3619738522 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08140feb20 0x7f08140fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:46.789 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.807+0000 7f081abc0700 1 --2- 172.21.15.88:0/3619738522 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08140ffd10 0x7f0814100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:46.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.807+0000 7f081abc0700 1 -- 172.21.15.88:0/3619738522 >> 172.21.15.88:0/3619738522 conn(0x7f08140fa100 msgr2=0x7f08140fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:46.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.807+0000 7f081abc0700 1 -- 172.21.15.88:0/3619738522 shutdown_connections 2023-07-14T17:44:46.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.807+0000 7f081abc0700 1 -- 172.21.15.88:0/3619738522 wait complete. 2023-07-14T17:44:46.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.807+0000 7f081abc0700 1 Processor -- start 2023-07-14T17:44:46.790 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.807+0000 7f081abc0700 1 -- start start 2023-07-14T17:44:46.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.807+0000 7f081abc0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08140feb20 0x7f0814069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:46.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.807+0000 7f081abc0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08140ffd10 0x7f081419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:46.793 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.807+0000 7f081abc0700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08141006a0 0x7f081419cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:46.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.807+0000 7f081abc0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081419d3c0 con 0x7f08140ffd10 2023-07-14T17:44:46.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.807+0000 7f081abc0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081419d530 con 0x7f08140feb20 2023-07-14T17:44:46.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.807+0000 7f081abc0700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f081419d6f0 con 0x7f08141006a0 2023-07-14T17:44:46.794 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.807+0000 7f0813fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08140ffd10 0x7f081419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:46.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.807+0000 7f0813fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08140ffd10 0x7f081419a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43646/0 (socket says 172.21.15.88:43646) 2023-07-14T17:44:46.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.808+0000 7f0813fff700 1 -- 172.21.15.88:0/3283158815 learned_addr learned my addr 172.21.15.88:0/3283158815 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:46.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.808+0000 7f081915d700 1 --2- 172.21.15.88:0/3283158815 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08141006a0 0x7f081419cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:46.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.808+0000 7f081895c700 1 --2- 172.21.15.88:0/3283158815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08140feb20 0x7f0814069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:46.795 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.808+0000 7f0813fff700 1 -- 172.21.15.88:0/3283158815 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08141006a0 msgr2=0x7f081419cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:46.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.808+0000 7f0813fff700 1 --2- 172.21.15.88:0/3283158815 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08141006a0 0x7f081419cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:46.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.808+0000 7f0813fff700 1 -- 172.21.15.88:0/3283158815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08140feb20 msgr2=0x7f0814069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:46.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.808+0000 7f0813fff700 1 --2- 172.21.15.88:0/3283158815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08140feb20 0x7f0814069340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:46.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.808+0000 7f0813fff700 1 -- 172.21.15.88:0/3283158815 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c009d00 con 0x7f08140ffd10 2023-07-14T17:44:46.796 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.808+0000 7f0813fff700 1 --2- 172.21.15.88:0/3283158815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08140ffd10 0x7f081419a5e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f080000c920 tx=0x7f080000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:46.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.808+0000 7f0811ffb700 1 -- 172.21.15.88:0/3283158815 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800007b00 con 0x7f08140ffd10 2023-07-14T17:44:46.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.808+0000 7f0811ffb700 1 -- 172.21.15.88:0/3283158815 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f080000ce70 con 0x7f08140ffd10 2023-07-14T17:44:46.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.808+0000 7f0811ffb700 1 -- 172.21.15.88:0/3283158815 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0800005440 con 0x7f08140ffd10 2023-07-14T17:44:46.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.808+0000 7f081abc0700 1 -- 172.21.15.88:0/3283158815 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f081419d9d0 con 0x7f08140ffd10 2023-07-14T17:44:46.797 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.808+0000 7f081abc0700 1 -- 172.21.15.88:0/3283158815 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f081419de90 con 0x7f08140ffd10 2023-07-14T17:44:46.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.809+0000 7f081abc0700 1 -- 172.21.15.88:0/3283158815 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0814105ff0 con 0x7f08140ffd10 2023-07-14T17:44:46.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.810+0000 7f0811ffb700 1 -- 172.21.15.88:0/3283158815 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f08000055a0 con 0x7f08140ffd10 2023-07-14T17:44:46.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.811+0000 7f0811ffb700 1 --2- 172.21.15.88:0/3283158815 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f08040a1220 0x7f08040a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:46.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.811+0000 7f081895c700 1 --2- 172.21.15.88:0/3283158815 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f08040a1220 0x7f08040a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:46.798 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.811+0000 7f0811ffb700 1 -- 172.21.15.88:0/3283158815 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f08000cca20 con 0x7f08140ffd10 2023-07-14T17:44:46.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.811+0000 7f081895c700 1 --2- 172.21.15.88:0/3283158815 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f08040a1220 0x7f08040a36d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0808005fd0 tx=0x7f0808005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:46.799 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:46.815+0000 7f0811ffb700 1 -- 172.21.15.88:0/3283158815 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0800095c60 con 0x7f08140ffd10 2023-07-14T17:44:47.026 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.045+0000 7f081abc0700 1 -- 172.21.15.88:0/3283158815 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f081402c8e0 con 0x7f08140ffd10 2023-07-14T17:44:47.027 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.046+0000 7f0811ffb700 1 -- 172.21.15.88:0/3283158815 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10885 (secure 0 0 0) 0x7f0800084020 con 0x7f08140ffd10 2023-07-14T17:44:47.028 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:47.028 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/2870142745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":2870142745},{"type":"v1","addr":"172.21.15.88:6835","nonce":2870142745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/783713410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":783713410},{"type":"v1","addr":"172.21.15.88:6837","nonce":783713410}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T17:42:42.314442+0000","modified":"2023-07-14T17:44:35.037715+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24506,"mds_1":24473,"mds_2":24469,"mds_3":24454,"mds_4":24449},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14616":{"gid":14616,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6839/3046592155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":3046592155},{"type":"v1","addr":"172.21.15.88:6839","nonce":3046592155}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14628":{"gid":14628,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6841/2509502918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6840","nonce":2509502918},{"type":"v1","addr":"172.21.15.88:6841","nonce":2509502918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24415":{"gid":24415,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6833/1716372869","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6832","nonce":1716372869},{"type":"v1","addr":"172.21.15.178:6833","nonce":1716372869}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24419":{"gid":24419,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.107:6833/2946270337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6832","nonce":2946270337},{"type":"v1","addr":"172.21.15.107:6833","nonce":2946270337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.178:6835/2332025068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6834","nonce":2332025068},{"type":"v1","addr":"172.21.15.178:6835","nonce":2332025068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24449":{"gid":24449,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.107:6835/3062637672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6834","nonce":3062637672},{"type":"v1","addr":"172.21.15.107:6835","nonce":3062637672}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.178:6837/3950404084","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6836","nonce":3950404084},{"type":"v1","addr":"172.21.15.178:6837","nonce":3950404084}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.178:6839/2953415077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6838","nonce":2953415077},{"type":"v1","addr":"172.21.15.178:6839","nonce":2953415077}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24473":{"gid":24473,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.107:6837/1152608615","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6836","nonce":1152608615},{"type":"v1","addr":"172.21.15.107:6837","nonce":1152608615}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24506":{"gid":24506,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.107:6839/31646548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6838","nonce":31646548},{"type":"v1","addr":"172.21.15.107:6839","nonce":31646548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T17:44:47.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.048+0000 7f081abc0700 1 -- 172.21.15.88:0/3283158815 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f08040a1220 msgr2=0x7f08040a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:47.029 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.048+0000 7f081abc0700 1 --2- 172.21.15.88:0/3283158815 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f08040a1220 0x7f08040a36d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0808005fd0 tx=0x7f0808005e50 comp rx=0 tx=0).stop 2023-07-14T17:44:47.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.048+0000 7f081abc0700 1 -- 172.21.15.88:0/3283158815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08140ffd10 msgr2=0x7f081419a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:47.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.048+0000 7f081abc0700 1 --2- 172.21.15.88:0/3283158815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08140ffd10 0x7f081419a5e0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f080000c920 tx=0x7f080000cc30 comp rx=0 tx=0).stop 2023-07-14T17:44:47.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.048+0000 7f081abc0700 1 -- 172.21.15.88:0/3283158815 shutdown_connections 2023-07-14T17:44:47.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.048+0000 7f081abc0700 1 --2- 172.21.15.88:0/3283158815 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f08141006a0 0x7f081419cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.030 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.048+0000 7f081abc0700 1 --2- 172.21.15.88:0/3283158815 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f08140feb20 0x7f0814069340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.048+0000 7f081abc0700 1 --2- 172.21.15.88:0/3283158815 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f08040a1220 0x7f08040a36d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.048+0000 7f081abc0700 1 --2- 172.21.15.88:0/3283158815 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f08140ffd10 0x7f081419a5e0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.048+0000 7f081abc0700 1 -- 172.21.15.88:0/3283158815 >> 172.21.15.88:0/3283158815 conn(0x7f08140fa100 msgr2=0x7f0814103130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:47.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.048+0000 7f081abc0700 1 -- 172.21.15.88:0/3283158815 shutdown_connections 2023-07-14T17:44:47.031 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.048+0000 7f081abc0700 1 -- 172.21.15.88:0/3283158815 wait complete. 2023-07-14T17:44:47.032 INFO:teuthology.orchestra.run.smithi088.stderr:dumped fsmap epoch 28 2023-07-14T17:44:47.042 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T17:44:47.164 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.183+0000 7f8d65462700 1 -- 172.21.15.88:0/2639683725 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d600fea90 msgr2=0x7f8d600feea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:47.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.183+0000 7f8d65462700 1 --2- 172.21.15.88:0/2639683725 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d600fea90 0x7f8d600feea0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8d50009990 tx=0x7f8d50009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.183+0000 7f8d65462700 1 -- 172.21.15.88:0/2639683725 shutdown_connections 2023-07-14T17:44:47.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.183+0000 7f8d65462700 1 --2- 172.21.15.88:0/2639683725 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d60100610 0x7f8d60106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.165 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.183+0000 7f8d65462700 1 --2- 172.21.15.88:0/2639683725 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d600fea90 0x7f8d600feea0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.183+0000 7f8d65462700 1 --2- 172.21.15.88:0/2639683725 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d600ffc80 0x7f8d601000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.183+0000 7f8d65462700 1 -- 172.21.15.88:0/2639683725 >> 172.21.15.88:0/2639683725 conn(0x7f8d600fa090 msgr2=0x7f8d600fc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:47.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.183+0000 7f8d65462700 1 -- 172.21.15.88:0/2639683725 shutdown_connections 2023-07-14T17:44:47.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.183+0000 7f8d65462700 1 -- 172.21.15.88:0/2639683725 wait complete. 2023-07-14T17:44:47.166 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.183+0000 7f8d65462700 1 Processor -- start 2023-07-14T17:44:47.167 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.183+0000 7f8d65462700 1 -- start start 2023-07-14T17:44:47.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d65462700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d600fea90 0x7f8d60102a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:47.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d65462700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d600ffc80 0x7f8d60101060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:47.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d65462700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d60100610 0x7f8d601015a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:47.169 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d65462700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d60101d40 con 0x7f8d600ffc80 2023-07-14T17:44:47.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d65462700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d60101eb0 con 0x7f8d600fea90 2023-07-14T17:44:47.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d65462700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d6019f860 con 0x7f8d60100610 2023-07-14T17:44:47.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d5e7fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d600ffc80 0x7f8d60101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:47.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d5e7fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d600ffc80 0x7f8d60101060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43660/0 (socket says 172.21.15.88:43660) 2023-07-14T17:44:47.170 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d5e7fc700 1 -- 172.21.15.88:0/2073518436 learned_addr learned my addr 172.21.15.88:0/2073518436 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:47.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d5effd700 1 --2- 172.21.15.88:0/2073518436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d600fea90 0x7f8d60102a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:47.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d5f7fe700 1 --2- 172.21.15.88:0/2073518436 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d60100610 0x7f8d601015a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:47.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d5e7fc700 1 -- 172.21.15.88:0/2073518436 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d60100610 msgr2=0x7f8d601015a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:47.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d5e7fc700 1 --2- 172.21.15.88:0/2073518436 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d60100610 0x7f8d601015a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.171 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d5e7fc700 1 -- 172.21.15.88:0/2073518436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d600fea90 msgr2=0x7f8d60102a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:47.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d5e7fc700 1 --2- 172.21.15.88:0/2073518436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d600fea90 0x7f8d60102a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d5e7fc700 1 -- 172.21.15.88:0/2073518436 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d500095f0 con 0x7f8d600ffc80 2023-07-14T17:44:47.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d5e7fc700 1 --2- 172.21.15.88:0/2073518436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d600ffc80 0x7f8d60101060 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8d5400d980 tx=0x7f8d5400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:47.172 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d47fff700 1 -- 172.21.15.88:0/2073518436 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d54004a30 con 0x7f8d600ffc80 2023-07-14T17:44:47.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d65462700 1 -- 172.21.15.88:0/2073518436 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d6019fb40 con 0x7f8d600ffc80 2023-07-14T17:44:47.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d47fff700 1 -- 172.21.15.88:0/2073518436 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8d540054b0 con 0x7f8d600ffc80 2023-07-14T17:44:47.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d47fff700 1 -- 172.21.15.88:0/2073518436 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5400fab0 con 0x7f8d600ffc80 2023-07-14T17:44:47.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.184+0000 7f8d65462700 1 -- 172.21.15.88:0/2073518436 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d601a0050 con 0x7f8d600ffc80 2023-07-14T17:44:47.173 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.185+0000 7f8d65462700 1 -- 172.21.15.88:0/2073518436 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d6010a130 con 0x7f8d600ffc80 2023-07-14T17:44:47.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.186+0000 7f8d47fff700 1 -- 172.21.15.88:0/2073518436 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8d5400fc10 con 0x7f8d600ffc80 2023-07-14T17:44:47.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.187+0000 7f8d47fff700 1 --2- 172.21.15.88:0/2073518436 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8d480a1220 0x7f8d480a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:47.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.187+0000 7f8d5effd700 1 --2- 172.21.15.88:0/2073518436 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8d480a1220 0x7f8d480a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:47.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.187+0000 7f8d47fff700 1 -- 172.21.15.88:0/2073518436 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f8d540cd0b0 con 0x7f8d600ffc80 2023-07-14T17:44:47.174 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.187+0000 7f8d5effd700 1 --2- 172.21.15.88:0/2073518436 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8d480a1220 0x7f8d480a36d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000b5c0 tx=0x7f8d50005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:47.175 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.190+0000 7f8d47fff700 1 -- 172.21.15.88:0/2073518436 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8d540962f0 con 0x7f8d600ffc80 2023-07-14T17:44:47.360 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.379+0000 7f8d65462700 1 -- 172.21.15.88:0/2073518436 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d601a0600 con 0x7f8d600ffc80 2023-07-14T17:44:47.361 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.380+0000 7f8d47fff700 1 -- 172.21.15.88:0/2073518436 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24607 (secure 0 0 0) 0x7f8d54017020 con 0x7f8d600ffc80 2023-07-14T17:44:47.362 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T17:44:47.362 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":90,"fsid":"b462a556-226c-11ee-9b33-001a4aab830c","created":"2023-07-14T17:35:48.898032+0000","modified":"2023-07-14T17:44:45.082655+0000","last_up_change":"2023-07-14T17:42:01.826711+0000","last_in_change":"2023-07-14T17:41:43.766597+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T17:38:50.126535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T17:42:40.725870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T17:42:41.512749+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T17:42:44.764573+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"32bbe236-4df9-42d6-8541-5d24d944eeee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6803","nonce":826983341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6805","nonce":826983341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6809","nonce":826983341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":826983341},{"type":"v1","addr":"172.21.15.88:6807","nonce":826983341}]},"public_addr":"172.21.15.88:6803/826983341","cluster_addr":"172.21.15.88:6805/826983341","heartbeat_back_addr":"172.21.15.88:6809/826983341","heartbeat_front_addr":"172.21.15.88:6807/826983341","state":["exists","up"]},{"osd":1,"uuid":"7d11240d-b997-40ba-b67b-f39d92dce9b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6801","nonce":1795455333}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6803","nonce":1795455333}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6807","nonce":1795455333}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1795455333},{"type":"v1","addr":"172.21.15.107:6805","nonce":1795455333}]},"public_addr":"172.21.15.107:6801/1795455333","cluster_addr":"172.21.15.107:6803/1795455333","heartbeat_back_addr":"172.21.15.107:6807/1795455333","heartbeat_front_addr":"172.21.15.107:6805/1795455333","state":["exists","up"]},{"osd":2,"uuid":"41542091-8564-496c-9b43-a827f60ac844","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6800","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6801","nonce":4021325462}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6802","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6803","nonce":4021325462}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6806","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6807","nonce":4021325462}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6804","nonce":4021325462},{"type":"v1","addr":"172.21.15.178:6805","nonce":4021325462}]},"public_addr":"172.21.15.178:6801/4021325462","cluster_addr":"172.21.15.178:6803/4021325462","heartbeat_back_addr":"172.21.15.178:6807/4021325462","heartbeat_front_addr":"172.21.15.178:6805/4021325462","state":["exists","up"]},{"osd":3,"uuid":"77c095e1-8805-435b-bb0e-1ff3ef4ba0a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6811","nonce":3836966618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6813","nonce":3836966618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6817","nonce":3836966618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":3836966618},{"type":"v1","addr":"172.21.15.88:6815","nonce":3836966618}]},"public_addr":"172.21.15.88:6811/3836966618","cluster_addr":"172.21.15.88:6813/3836966618","heartbeat_back_addr":"172.21.15.88:6817/3836966618","heartbeat_front_addr":"172.21.15.88:6815/3836966618","state":["exists","up"]},{"osd":4,"uuid":"ae0f806e-a7ef-445d-a7d4-14823157b841","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6809","nonce":2380989846}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6811","nonce":2380989846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6815","nonce":2380989846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":2380989846},{"type":"v1","addr":"172.21.15.107:6813","nonce":2380989846}]},"public_addr":"172.21.15.107:6809/2380989846","cluster_addr":"172.21.15.107:6811/2380989846","heartbeat_back_addr":"172.21.15.107:6815/2380989846","heartbeat_front_addr":"172.21.15.107:6813/2380989846","state":["exists","up"]},{"osd":5,"uuid":"b926d4fa-a9de-4bb6-8e00-30a2b8a35b36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6808","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6809","nonce":4006439725}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6810","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6811","nonce":4006439725}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6814","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6815","nonce":4006439725}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6812","nonce":4006439725},{"type":"v1","addr":"172.21.15.178:6813","nonce":4006439725}]},"public_addr":"172.21.15.178:6809/4006439725","cluster_addr":"172.21.15.178:6811/4006439725","heartbeat_back_addr":"172.21.15.178:6815/4006439725","heartbeat_front_addr":"172.21.15.178:6813/4006439725","state":["exists","up"]},{"osd":6,"uuid":"206cd369-2456-41c5-8a3f-d1efce5bb45a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6819","nonce":1424514128}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6821","nonce":1424514128}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6825","nonce":1424514128}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1424514128},{"type":"v1","addr":"172.21.15.88:6823","nonce":1424514128}]},"public_addr":"172.21.15.88:6819/1424514128","cluster_addr":"172.21.15.88:6821/1424514128","heartbeat_back_addr":"172.21.15.88:6825/1424514128","heartbeat_front_addr":"172.21.15.88:6823/1424514128","state":["exists","up"]},{"osd":7,"uuid":"59b9919f-52fb-42b6-b1d4-dee1df75c23a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6817","nonce":2214437741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6819","nonce":2214437741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6823","nonce":2214437741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":2214437741},{"type":"v1","addr":"172.21.15.107:6821","nonce":2214437741}]},"public_addr":"172.21.15.107:6817/2214437741","cluster_addr":"172.21.15.107:6819/2214437741","heartbeat_back_addr":"172.21.15.107:6823/2214437741","heartbeat_front_addr":"172.21.15.107:6821/2214437741","state":["exists","up"]},{"osd":8,"uuid":"81ec848e-9acc-40e8-8890-3d80f6844597","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6816","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6817","nonce":2305567673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6818","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6819","nonce":2305567673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6822","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6823","nonce":2305567673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6820","nonce":2305567673},{"type":"v1","addr":"172.21.15.178:6821","nonce":2305567673}]},"public_addr":"172.21.15.178:6817/2305567673","cluster_addr":"172.21.15.178:6819/2305567673","heartbeat_back_addr":"172.21.15.178:6823/2305567673","heartbeat_front_addr":"172.21.15.178:6821/2305567673","state":["exists","up"]},{"osd":9,"uuid":"96a3d388-7f40-40ba-80ac-b670a13a9993","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6827","nonce":3211834989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6829","nonce":3211834989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6833","nonce":3211834989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":3211834989},{"type":"v1","addr":"172.21.15.88:6831","nonce":3211834989}]},"public_addr":"172.21.15.88:6827/3211834989","cluster_addr":"172.21.15.88:6829/3211834989","heartbeat_back_addr":"172.21.15.88:6833/3211834989","heartbeat_front_addr":"172.21.15.88:6831/3211834989","state":["exists","up"]},{"osd":10,"uuid":"bc27ef27-7791-4c70-be0b-84e60b24af36","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6825","nonce":1223599278}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6827","nonce":1223599278}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6831","nonce":1223599278}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1223599278},{"type":"v1","addr":"172.21.15.107:6829","nonce":1223599278}]},"public_addr":"172.21.15.107:6825/1223599278","cluster_addr":"172.21.15.107:6827/1223599278","heartbeat_back_addr":"172.21.15.107:6831/1223599278","heartbeat_front_addr":"172.21.15.107:6829/1223599278","state":["exists","up"]},{"osd":11,"uuid":"ba0e783d-7fbe-420c-93d9-81923f17bb9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":77,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6824","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6825","nonce":1862417235}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6826","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6827","nonce":1862417235}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6830","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6831","nonce":1862417235}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.178:6828","nonce":1862417235},{"type":"v1","addr":"172.21.15.178:6829","nonce":1862417235}]},"public_addr":"172.21.15.178:6825/1862417235","cluster_addr":"172.21.15.178:6827/1862417235","heartbeat_back_addr":"172.21.15.178:6831/1862417235","heartbeat_front_addr":"172.21.15.178:6829/1862417235","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:05.539973+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:26.822508+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:38:48.345005+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:10.178959+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:31.475411+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:39:52.806252+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:13.793562+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:34.801435+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:40:56.215549+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:17.164275+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:41:38.445244+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T17:42:00.101507+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.6","mappings":[{"from":4,"to":6}]},{"pgid":"3.1e","mappings":[{"from":5,"to":8}]},{"pgid":"3.2c","mappings":[{"from":4,"to":9}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.6","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.88:0/1441439704":"2023-07-15T17:36:43.799741+0000","172.21.15.88:0/3897985004":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/2424184187":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6801/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:6800/1369181448":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/1790623516":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3232411191":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/3329866608":"2023-07-15T17:36:21.153428+0000","172.21.15.88:0/954461155":"2023-07-15T17:36:43.799741+0000","172.21.15.88:6800/2424184187":"2023-07-15T17:36:43.799741+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T17:44:47.363 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.382+0000 7f8d65462700 1 -- 172.21.15.88:0/2073518436 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8d480a1220 msgr2=0x7f8d480a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:47.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.382+0000 7f8d65462700 1 --2- 172.21.15.88:0/2073518436 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8d480a1220 0x7f8d480a36d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000b5c0 tx=0x7f8d50005960 comp rx=0 tx=0).stop 2023-07-14T17:44:47.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.382+0000 7f8d65462700 1 -- 172.21.15.88:0/2073518436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d600ffc80 msgr2=0x7f8d60101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:47.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.382+0000 7f8d65462700 1 --2- 172.21.15.88:0/2073518436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d600ffc80 0x7f8d60101060 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8d5400d980 tx=0x7f8d5400dc90 comp rx=0 tx=0).stop 2023-07-14T17:44:47.364 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.382+0000 7f8d65462700 1 -- 172.21.15.88:0/2073518436 shutdown_connections 2023-07-14T17:44:47.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.382+0000 7f8d65462700 1 --2- 172.21.15.88:0/2073518436 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f8d60100610 0x7f8d601015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.382+0000 7f8d65462700 1 --2- 172.21.15.88:0/2073518436 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f8d600fea90 0x7f8d60102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.382+0000 7f8d65462700 1 --2- 172.21.15.88:0/2073518436 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f8d480a1220 0x7f8d480a36d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.382+0000 7f8d65462700 1 --2- 172.21.15.88:0/2073518436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8d600ffc80 0x7f8d60101060 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.365 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.382+0000 7f8d65462700 1 -- 172.21.15.88:0/2073518436 >> 172.21.15.88:0/2073518436 conn(0x7f8d600fa090 msgr2=0x7f8d600677c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:47.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.382+0000 7f8d65462700 1 -- 172.21.15.88:0/2073518436 shutdown_connections 2023-07-14T17:44:47.366 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.382+0000 7f8d65462700 1 -- 172.21.15.88:0/2073518436 wait complete. 2023-07-14T17:44:47.375 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:47 smithi088 ceph-mon[127465]: pgmap v321: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.9 KiB/s wr, 15 op/s 2023-07-14T17:44:47.375 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:47 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3781083763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:47.376 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:47 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3283158815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:47.379 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-07-14T17:44:47.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:47 smithi178 ceph-mon[129368]: pgmap v321: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.9 KiB/s wr, 15 op/s 2023-07-14T17:44:47.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:47 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3781083763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:47.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:47 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3283158815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:47.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-mon[128999]: pgmap v321: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 1.9 KiB/s wr, 15 op/s 2023-07-14T17:44:47.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3781083763' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:47.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3283158815' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T17:44:47.501 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.520+0000 7f4df9ac6700 1 -- 172.21.15.88:0/938838816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4df40ffd00 msgr2=0x7f4df4100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:47.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.520+0000 7f4df9ac6700 1 --2- 172.21.15.88:0/938838816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4df40ffd00 0x7f4df4100150 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f4de8009990 tx=0x7f4de8009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.520+0000 7f4df9ac6700 1 -- 172.21.15.88:0/938838816 shutdown_connections 2023-07-14T17:44:47.502 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.520+0000 7f4df9ac6700 1 --2- 172.21.15.88:0/938838816 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df4100690 0x7f4df4106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.520+0000 7f4df9ac6700 1 --2- 172.21.15.88:0/938838816 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4df40feb10 0x7f4df40fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.520+0000 7f4df9ac6700 1 --2- 172.21.15.88:0/938838816 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4df40ffd00 0x7f4df4100150 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.520+0000 7f4df9ac6700 1 -- 172.21.15.88:0/938838816 >> 172.21.15.88:0/938838816 conn(0x7f4df40fa130 msgr2=0x7f4df40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:47.503 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.520+0000 7f4df9ac6700 1 -- 172.21.15.88:0/938838816 shutdown_connections 2023-07-14T17:44:47.504 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.520+0000 7f4df9ac6700 1 -- 172.21.15.88:0/938838816 wait complete. 2023-07-14T17:44:47.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.521+0000 7f4df9ac6700 1 Processor -- start 2023-07-14T17:44:47.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.521+0000 7f4df9ac6700 1 -- start start 2023-07-14T17:44:47.506 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.521+0000 7f4df9ac6700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df40feb10 0x7f4df4195810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:47.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.521+0000 7f4df9ac6700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4df40ffd00 0x7f4df4195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:47.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.521+0000 7f4df9ac6700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4df4100690 0x7f4df419cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:47.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.521+0000 7f4df9ac6700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df419d390 con 0x7f4df4100690 2023-07-14T17:44:47.507 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.521+0000 7f4df9ac6700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df419d4d0 con 0x7f4df40feb10 2023-07-14T17:44:47.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.521+0000 7f4df9ac6700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df419d660 con 0x7f4df40ffd00 2023-07-14T17:44:47.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.521+0000 7f4df3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4df4100690 0x7f4df419cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:47.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.521+0000 7f4df3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4df4100690 0x7f4df419cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43676/0 (socket says 172.21.15.88:43676) 2023-07-14T17:44:47.508 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.521+0000 7f4df3fff700 1 -- 172.21.15.88:0/3166269231 learned_addr learned my addr 172.21.15.88:0/3166269231 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:47.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.521+0000 7f4df2ffd700 1 --2- 172.21.15.88:0/3166269231 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4df40ffd00 0x7f4df4195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:47.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.521+0000 7f4df37fe700 1 --2- 172.21.15.88:0/3166269231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df40feb10 0x7f4df4195810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:47.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.521+0000 7f4df3fff700 1 -- 172.21.15.88:0/3166269231 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4df40ffd00 msgr2=0x7f4df4195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:47.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.521+0000 7f4df3fff700 1 --2- 172.21.15.88:0/3166269231 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4df40ffd00 0x7f4df4195d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.509 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.521+0000 7f4df3fff700 1 -- 172.21.15.88:0/3166269231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df40feb10 msgr2=0x7f4df4195810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:47.510 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.521+0000 7f4df3fff700 1 --2- 172.21.15.88:0/3166269231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df40feb10 0x7f4df4195810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.511 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.521+0000 7f4df3fff700 1 -- 172.21.15.88:0/3166269231 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de80095f0 con 0x7f4df4100690 2023-07-14T17:44:47.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.522+0000 7f4df3fff700 1 --2- 172.21.15.88:0/3166269231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4df4100690 0x7f4df419cdc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00ada0 tx=0x7f4dec00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:47.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.522+0000 7f4df0ff9700 1 -- 172.21.15.88:0/3166269231 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec00f070 con 0x7f4df4100690 2023-07-14T17:44:47.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.522+0000 7f4df9ac6700 1 -- 172.21.15.88:0/3166269231 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df419d940 con 0x7f4df4100690 2023-07-14T17:44:47.512 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.522+0000 7f4df0ff9700 1 -- 172.21.15.88:0/3166269231 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4dec007b30 con 0x7f4df4100690 2023-07-14T17:44:47.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.522+0000 7f4df0ff9700 1 -- 172.21.15.88:0/3166269231 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dec0053e0 con 0x7f4df4100690 2023-07-14T17:44:47.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.522+0000 7f4df9ac6700 1 -- 172.21.15.88:0/3166269231 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df419de00 con 0x7f4df4100690 2023-07-14T17:44:47.513 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.523+0000 7f4df9ac6700 1 -- 172.21.15.88:0/3166269231 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df4105a90 con 0x7f4df4100690 2023-07-14T17:44:47.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.524+0000 7f4df0ff9700 1 -- 172.21.15.88:0/3166269231 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4dec00c980 con 0x7f4df4100690 2023-07-14T17:44:47.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.524+0000 7f4df0ff9700 1 --2- 172.21.15.88:0/3166269231 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4ddc0a11e0 0x7f4ddc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:47.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.524+0000 7f4df0ff9700 1 -- 172.21.15.88:0/3166269231 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9078+0+0 (secure 0 0 0) 0x7f4dec0ccb50 con 0x7f4df4100690 2023-07-14T17:44:47.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.524+0000 7f4df37fe700 1 --2- 172.21.15.88:0/3166269231 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4ddc0a11e0 0x7f4ddc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:47.514 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.525+0000 7f4df37fe700 1 --2- 172.21.15.88:0/3166269231 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4ddc0a11e0 0x7f4ddc0a3690 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4de4005fd0 tx=0x7f4de4005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:47.515 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.528+0000 7f4df0ff9700 1 -- 172.21.15.88:0/3166269231 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4dec095e80 con 0x7f4df4100690 2023-07-14T17:44:47.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.762+0000 7f4df9ac6700 1 -- 172.21.15.88:0/3166269231 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4df402cb10 con 0x7f4df4100690 2023-07-14T17:44:47.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.776+0000 7f4df0ff9700 1 -- 172.21.15.88:0/3166269231 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v29) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4dec00cc30 con 0x7f4df4100690 2023-07-14T17:44:47.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.778+0000 7f4df9ac6700 1 -- 172.21.15.88:0/3166269231 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4ddc0a11e0 msgr2=0x7f4ddc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:47.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.778+0000 7f4df9ac6700 1 --2- 172.21.15.88:0/3166269231 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4ddc0a11e0 0x7f4ddc0a3690 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4de4005fd0 tx=0x7f4de4005e50 comp rx=0 tx=0).stop 2023-07-14T17:44:47.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.778+0000 7f4df9ac6700 1 -- 172.21.15.88:0/3166269231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4df4100690 msgr2=0x7f4df419cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:47.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.778+0000 7f4df9ac6700 1 --2- 172.21.15.88:0/3166269231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4df4100690 0x7f4df419cdc0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00ada0 tx=0x7f4dec00c330 comp rx=0 tx=0).stop 2023-07-14T17:44:47.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.778+0000 7f4df9ac6700 1 -- 172.21.15.88:0/3166269231 shutdown_connections 2023-07-14T17:44:47.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.778+0000 7f4df9ac6700 1 --2- 172.21.15.88:0/3166269231 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f4ddc0a11e0 0x7f4ddc0a3690 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.778+0000 7f4df9ac6700 1 --2- 172.21.15.88:0/3166269231 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4df4100690 0x7f4df419cdc0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.761 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.778+0000 7f4df9ac6700 1 --2- 172.21.15.88:0/3166269231 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f4df40feb10 0x7f4df4195810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.778+0000 7f4df9ac6700 1 --2- 172.21.15.88:0/3166269231 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f4df40ffd00 0x7f4df4195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.778+0000 7f4df9ac6700 1 -- 172.21.15.88:0/3166269231 >> 172.21.15.88:0/3166269231 conn(0x7f4df40fa130 msgr2=0x7f4df40fc410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:47.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.778+0000 7f4df9ac6700 1 -- 172.21.15.88:0/3166269231 shutdown_connections 2023-07-14T17:44:47.762 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.778+0000 7f4df9ac6700 1 -- 172.21.15.88:0/3166269231 wait complete. 2023-07-14T17:44:47.763 INFO:teuthology.orchestra.run.smithi088.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-07-14T17:44:47.775 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-07-14T17:44:47.901 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.920+0000 7f0f474d8700 1 -- 172.21.15.88:0/2953739968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f401000b0 msgr2=0x7f0f401004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:47.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.920+0000 7f0f474d8700 1 --2- 172.21.15.88:0/2953739968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f401000b0 0x7f0f401004c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0f30009990 tx=0x7f0f30009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.920+0000 7f0f474d8700 1 -- 172.21.15.88:0/2953739968 shutdown_connections 2023-07-14T17:44:47.902 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.920+0000 7f0f474d8700 1 --2- 172.21.15.88:0/2953739968 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f400ff6b0 0x7f0f40106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.920+0000 7f0f474d8700 1 --2- 172.21.15.88:0/2953739968 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f401000b0 0x7f0f401004c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.920+0000 7f0f474d8700 1 --2- 172.21.15.88:0/2953739968 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f400fed00 0x7f0f400ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.903 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.920+0000 7f0f474d8700 1 -- 172.21.15.88:0/2953739968 >> 172.21.15.88:0/2953739968 conn(0x7f0f400f98a0 msgr2=0x7f0f400fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:47.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.920+0000 7f0f474d8700 1 -- 172.21.15.88:0/2953739968 shutdown_connections 2023-07-14T17:44:47.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.920+0000 7f0f474d8700 1 -- 172.21.15.88:0/2953739968 wait complete. 2023-07-14T17:44:47.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f474d8700 1 Processor -- start 2023-07-14T17:44:47.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f474d8700 1 -- start start 2023-07-14T17:44:47.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f474d8700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f400fed00 0x7f0f40102ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:47.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f474d8700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f400ff6b0 0x7f0f401010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:47.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f474d8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f401000b0 0x7f0f40101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:47.907 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f474d8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f40101dd0 con 0x7f0f401000b0 2023-07-14T17:44:47.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f474d8700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f40101f40 con 0x7f0f400ff6b0 2023-07-14T17:44:47.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f474d8700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4019f8e0 con 0x7f0f400fed00 2023-07-14T17:44:47.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f45a75700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f401000b0 0x7f0f40101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:47.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f45a75700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f401000b0 0x7f0f40101630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43714/0 (socket says 172.21.15.88:43714) 2023-07-14T17:44:47.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f45a75700 1 -- 172.21.15.88:0/239019189 learned_addr learned my addr 172.21.15.88:0/239019189 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:47.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f45274700 1 --2- 172.21.15.88:0/239019189 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f400fed00 0x7f0f40102ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:47.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f44a73700 1 --2- 172.21.15.88:0/239019189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f400ff6b0 0x7f0f401010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:47.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f45a75700 1 -- 172.21.15.88:0/239019189 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f400fed00 msgr2=0x7f0f40102ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:47.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f45a75700 1 --2- 172.21.15.88:0/239019189 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f400fed00 0x7f0f40102ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f45a75700 1 -- 172.21.15.88:0/239019189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f400ff6b0 msgr2=0x7f0f401010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f45a75700 1 --2- 172.21.15.88:0/239019189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f400ff6b0 0x7f0f401010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f45a75700 1 -- 172.21.15.88:0/239019189 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f300095f0 con 0x7f0f401000b0 2023-07-14T17:44:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f44a73700 1 --2- 172.21.15.88:0/239019189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f400ff6b0 0x7f0f401010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T17:44:47.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f45a75700 1 --2- 172.21.15.88:0/239019189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f401000b0 0x7f0f40101630 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0f38007d70 tx=0x7f0f3800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.921+0000 7f0f367fc700 1 -- 172.21.15.88:0/239019189 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3800ad40 con 0x7f0f401000b0 2023-07-14T17:44:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.922+0000 7f0f367fc700 1 -- 172.21.15.88:0/239019189 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0f380128b0 con 0x7f0f401000b0 2023-07-14T17:44:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.922+0000 7f0f474d8700 1 -- 172.21.15.88:0/239019189 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f4019fbc0 con 0x7f0f401000b0 2023-07-14T17:44:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.922+0000 7f0f367fc700 1 -- 172.21.15.88:0/239019189 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f380115a0 con 0x7f0f401000b0 2023-07-14T17:44:47.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.922+0000 7f0f474d8700 1 -- 172.21.15.88:0/239019189 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f401a0080 con 0x7f0f401000b0 2023-07-14T17:44:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.922+0000 7f0f474d8700 1 -- 172.21.15.88:0/239019189 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f4010a160 con 0x7f0f401000b0 2023-07-14T17:44:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.923+0000 7f0f367fc700 1 -- 172.21.15.88:0/239019189 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f0f38012a20 con 0x7f0f401000b0 2023-07-14T17:44:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.924+0000 7f0f367fc700 1 --2- 172.21.15.88:0/239019189 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0f2c0a11e0 0x7f0f2c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.924+0000 7f0f45274700 1 --2- 172.21.15.88:0/239019189 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0f2c0a11e0 0x7f0f2c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:47.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.924+0000 7f0f367fc700 1 -- 172.21.15.88:0/239019189 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9508+0+0 (secure 0 0 0) 0x7f0f380cd0c0 con 0x7f0f401000b0 2023-07-14T17:44:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.924+0000 7f0f45274700 1 --2- 172.21.15.88:0/239019189 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0f2c0a11e0 0x7f0f2c0a3690 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0f3000b5c0 tx=0x7f0f30005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:47.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:47.928+0000 7f0f367fc700 1 -- 172.21.15.88:0/239019189 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0f380962c0 con 0x7f0f401000b0 2023-07-14T17:44:48.075 INFO:journalctl@ceph.mds.a.smithi088.stdout:Jul 14 17:44:47 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-a[162115]: ignoring --setuser ceph since I am not root 2023-07-14T17:44:48.076 INFO:journalctl@ceph.mds.a.smithi088.stdout:Jul 14 17:44:47 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-a[162115]: ignoring --setgroup ceph since I am not root 2023-07-14T17:44:48.076 INFO:journalctl@ceph.mds.a.smithi088.stdout:Jul 14 17:44:47 smithi088 ceph-mds[162139]: -- 172.21.15.88:0/821673485 <== mon.0 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b9f4c571e0 con 0x55b9f4c2cc00 2023-07-14T17:44:48.077 INFO:journalctl@ceph.mds.a.smithi088.stdout:Jul 14 17:44:47 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-a[162115]: starting mds.a at 2023-07-14T17:44:48.077 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:47 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mon-a[127441]: 2023-07-14T17:44:47.763+0000 7f90469e1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T17:44:48.078 INFO:journalctl@ceph.mds.g.smithi088.stdout:Jul 14 17:44:47 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-g[161729]: ignoring --setuser ceph since I am not root 2023-07-14T17:44:48.078 INFO:journalctl@ceph.mds.g.smithi088.stdout:Jul 14 17:44:47 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-g[161729]: ignoring --setgroup ceph since I am not root 2023-07-14T17:44:48.078 INFO:journalctl@ceph.mds.g.smithi088.stdout:Jul 14 17:44:47 smithi088 ceph-mds[161753]: -- 172.21.15.88:0/314960797 <== mon.0 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c5436d71e0 con 0x55c5436ad000 2023-07-14T17:44:48.078 INFO:journalctl@ceph.mds.g.smithi088.stdout:Jul 14 17:44:47 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-g[161729]: starting mds.g at 2023-07-14T17:44:48.079 INFO:journalctl@ceph.mds.e.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-e[153402]: ignoring --setuser ceph since I am not root 2023-07-14T17:44:48.079 INFO:journalctl@ceph.mds.e.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-e[153402]: ignoring --setgroup ceph since I am not root 2023-07-14T17:44:48.080 INFO:journalctl@ceph.mds.e.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-mds[153426]: -- 172.21.15.107:0/2096163441 <== mon.2 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561e233bf1e0 con 0x561e23394800 2023-07-14T17:44:48.080 INFO:journalctl@ceph.mds.e.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-e[153402]: starting mds.e at 2023-07-14T17:44:48.080 INFO:journalctl@ceph.mds.i.smithi178.stdout:Jul 14 17:44:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-i[155583]: ignoring --setuser ceph since I am not root 2023-07-14T17:44:48.080 INFO:journalctl@ceph.mds.i.smithi178.stdout:Jul 14 17:44:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-i[155583]: ignoring --setgroup ceph since I am not root 2023-07-14T17:44:48.081 INFO:journalctl@ceph.mds.i.smithi178.stdout:Jul 14 17:44:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-i[155583]: starting mds.i at 2023-07-14T17:44:48.081 INFO:journalctl@ceph.mds.h.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-h[153740]: ignoring --setuser ceph since I am not root 2023-07-14T17:44:48.081 INFO:journalctl@ceph.mds.h.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-h[153740]: ignoring --setgroup ceph since I am not root 2023-07-14T17:44:48.082 INFO:journalctl@ceph.mds.h.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-mds[153764]: -- 172.21.15.107:0/3614727536 <== mon.2 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555c79e2b1e0 con 0x555c79e00c00 2023-07-14T17:44:48.082 INFO:journalctl@ceph.mds.h.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-h[153740]: starting mds.h at 2023-07-14T17:44:48.082 INFO:journalctl@ceph.mds.f.smithi178.stdout:Jul 14 17:44:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-f[154891]: ignoring --setuser ceph since I am not root 2023-07-14T17:44:48.082 INFO:journalctl@ceph.mds.f.smithi178.stdout:Jul 14 17:44:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-f[154891]: ignoring --setgroup ceph since I am not root 2023-07-14T17:44:48.083 INFO:journalctl@ceph.mds.f.smithi178.stdout:Jul 14 17:44:47 smithi178 ceph-mds[154915]: -- 172.21.15.178:0/3788360198 <== mon.0 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562a24afd1e0 con 0x562a24ad2800 2023-07-14T17:44:48.083 INFO:journalctl@ceph.mds.f.smithi178.stdout:Jul 14 17:44:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-f[154891]: starting mds.f at 2023-07-14T17:44:48.083 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-b[154079]: -7986> 2023-07-14T17:43:07.891+0000 7fb592e7d700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-07-14T17:44:48.084 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-b[154079]: ignoring --setuser ceph since I am not root 2023-07-14T17:44:48.084 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-b[154079]: ignoring --setgroup ceph since I am not root 2023-07-14T17:44:48.084 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-mds[154103]: -- 172.21.15.107:0/2615013218 <== mon.2 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555ec9b5f1e0 con 0x555ec9b35000 2023-07-14T17:44:48.084 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-b[154079]: starting mds.b at 2023-07-14T17:44:48.085 INFO:journalctl@ceph.mds.l.smithi178.stdout:Jul 14 17:44:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-l[154546]: ignoring --setuser ceph since I am not root 2023-07-14T17:44:48.085 INFO:journalctl@ceph.mds.l.smithi178.stdout:Jul 14 17:44:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-l[154546]: ignoring --setgroup ceph since I am not root 2023-07-14T17:44:48.085 INFO:journalctl@ceph.mds.l.smithi178.stdout:Jul 14 17:44:47 smithi178 ceph-mds[154570]: -- 172.21.15.178:0/3364569348 <== mon.1 v2:172.21.15.178:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5564806f91e0 con 0x5564806ce800 2023-07-14T17:44:48.085 INFO:journalctl@ceph.mds.l.smithi178.stdout:Jul 14 17:44:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-l[154546]: starting mds.l at 2023-07-14T17:44:48.085 INFO:journalctl@ceph.mds.k.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-k[153063]: ignoring --setuser ceph since I am not root 2023-07-14T17:44:48.086 INFO:journalctl@ceph.mds.k.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-k[153063]: ignoring --setgroup ceph since I am not root 2023-07-14T17:44:48.086 INFO:journalctl@ceph.mds.k.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-mds[153087]: -- 172.21.15.107:0/2681829184 <== mon.2 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556281c331e0 con 0x556281c08800 2023-07-14T17:44:48.086 INFO:journalctl@ceph.mds.k.smithi107.stdout:Jul 14 17:44:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-k[153063]: starting mds.k at 2023-07-14T17:44:48.086 INFO:journalctl@ceph.mds.c.smithi178.stdout:Jul 14 17:44:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-c[155243]: ignoring --setuser ceph since I am not root 2023-07-14T17:44:48.087 INFO:journalctl@ceph.mds.c.smithi178.stdout:Jul 14 17:44:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-c[155243]: ignoring --setgroup ceph since I am not root 2023-07-14T17:44:48.087 INFO:journalctl@ceph.mds.c.smithi178.stdout:Jul 14 17:44:47 smithi178 ceph-mds[155268]: -- 172.21.15.178:0/4154206513 <== mon.2 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560417e711e0 con 0x560417e46800 2023-07-14T17:44:48.087 INFO:journalctl@ceph.mds.c.smithi178.stdout:Jul 14 17:44:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-c[155243]: starting mds.c at 2023-07-14T17:44:48.143 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.162+0000 7f0f474d8700 1 -- 172.21.15.88:0/239019189 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0f401a0690 con 0x7f0f401000b0 2023-07-14T17:44:48.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:48 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2073518436' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:48.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:48 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3166269231' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T17:44:48.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:48 smithi178 ceph-mon[129368]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T17:44:48.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:48 smithi178 ceph-mon[129368]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T17:44:48.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:48 smithi178 ceph-mon[129368]: osdmap e91: 12 total, 12 up, 12 in 2023-07-14T17:44:48.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:48 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3166269231' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T17:44:48.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:48 smithi178 ceph-mon[129368]: fsmap cephfs:0/5 2 up:standby, 5 failed 2023-07-14T17:44:48.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:48 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2073518436' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:48.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:48 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3166269231' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T17:44:48.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:48 smithi088 ceph-mon[127465]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T17:44:48.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:48 smithi088 ceph-mon[127465]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T17:44:48.443 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:48 smithi088 ceph-mon[127465]: osdmap e91: 12 total, 12 up, 12 in 2023-07-14T17:44:48.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:48 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3166269231' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T17:44:48.444 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:48 smithi088 ceph-mon[127465]: fsmap cephfs:0/5 2 up:standby, 5 failed 2023-07-14T17:44:48.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:48 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2073518436' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T17:44:48.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:48 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3166269231' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T17:44:48.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:48 smithi107 ceph-mon[128999]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T17:44:48.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:48 smithi107 ceph-mon[128999]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T17:44:48.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:48 smithi107 ceph-mon[128999]: osdmap e91: 12 total, 12 up, 12 in 2023-07-14T17:44:48.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:48 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3166269231' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T17:44:48.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:48 smithi107 ceph-mon[128999]: fsmap cephfs:0/5 2 up:standby, 5 failed 2023-07-14T17:44:48.753 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.772+0000 7f0f367fc700 1 -- 172.21.15.88:0/239019189 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v30) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0f380998e0 con 0x7f0f401000b0 2023-07-14T17:44:48.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.776+0000 7f0f474d8700 1 -- 172.21.15.88:0/239019189 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0f2c0a11e0 msgr2=0x7f0f2c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:48.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.776+0000 7f0f474d8700 1 --2- 172.21.15.88:0/239019189 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0f2c0a11e0 0x7f0f2c0a3690 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0f3000b5c0 tx=0x7f0f30005fb0 comp rx=0 tx=0).stop 2023-07-14T17:44:48.757 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.776+0000 7f0f474d8700 1 -- 172.21.15.88:0/239019189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f401000b0 msgr2=0x7f0f40101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:48.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.776+0000 7f0f474d8700 1 --2- 172.21.15.88:0/239019189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f401000b0 0x7f0f40101630 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0f38007d70 tx=0x7f0f3800a340 comp rx=0 tx=0).stop 2023-07-14T17:44:48.758 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.777+0000 7f0f474d8700 1 -- 172.21.15.88:0/239019189 shutdown_connections 2023-07-14T17:44:48.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.777+0000 7f0f474d8700 1 --2- 172.21.15.88:0/239019189 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f0f2c0a11e0 0x7f0f2c0a3690 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:48.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.777+0000 7f0f474d8700 1 --2- 172.21.15.88:0/239019189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f401000b0 0x7f0f40101630 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:48.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.777+0000 7f0f474d8700 1 --2- 172.21.15.88:0/239019189 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f0f400fed00 0x7f0f40102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:48.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.777+0000 7f0f474d8700 1 --2- 172.21.15.88:0/239019189 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f0f400ff6b0 0x7f0f401010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:48.759 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.777+0000 7f0f474d8700 1 -- 172.21.15.88:0/239019189 >> 172.21.15.88:0/239019189 conn(0x7f0f400f98a0 msgr2=0x7f0f400fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:48.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.777+0000 7f0f474d8700 1 -- 172.21.15.88:0/239019189 shutdown_connections 2023-07-14T17:44:48.760 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.778+0000 7f0f474d8700 1 -- 172.21.15.88:0/239019189 wait complete. 2023-07-14T17:44:48.773 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-07-14T17:44:48.904 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.923+0000 7fc4dfbfd700 1 -- 172.21.15.88:0/1390039278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4d80feb20 msgr2=0x7fc4d80fef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:48.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.923+0000 7fc4dfbfd700 1 --2- 172.21.15.88:0/1390039278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4d80feb20 0x7fc4d80fef30 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4009990 tx=0x7fc4d4009ca0 comp rx=0 tx=0).stop 2023-07-14T17:44:48.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.923+0000 7fc4dfbfd700 1 -- 172.21.15.88:0/1390039278 shutdown_connections 2023-07-14T17:44:48.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.923+0000 7fc4dfbfd700 1 --2- 172.21.15.88:0/1390039278 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4d81006a0 0x7fc4d8106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:48.905 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.923+0000 7fc4dfbfd700 1 --2- 172.21.15.88:0/1390039278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4d80feb20 0x7fc4d80fef30 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:48.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.923+0000 7fc4dfbfd700 1 --2- 172.21.15.88:0/1390039278 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc4d80ffd10 0x7fc4d8100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:48.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.923+0000 7fc4dfbfd700 1 -- 172.21.15.88:0/1390039278 >> 172.21.15.88:0/1390039278 conn(0x7fc4d80fa160 msgr2=0x7fc4d80fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:48.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.923+0000 7fc4dfbfd700 1 -- 172.21.15.88:0/1390039278 shutdown_connections 2023-07-14T17:44:48.906 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.923+0000 7fc4dfbfd700 1 -- 172.21.15.88:0/1390039278 wait complete. 2023-07-14T17:44:48.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.923+0000 7fc4dfbfd700 1 Processor -- start 2023-07-14T17:44:48.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.924+0000 7fc4dfbfd700 1 -- start start 2023-07-14T17:44:48.908 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.924+0000 7fc4dfbfd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4d80feb20 0x7fc4d8069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:48.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.924+0000 7fc4dfbfd700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4d80ffd10 0x7fc4d819a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:48.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.924+0000 7fc4dfbfd700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc4d81006a0 0x7fc4d819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:48.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.924+0000 7fc4dfbfd700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d819d400 con 0x7fc4d80feb20 2023-07-14T17:44:48.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.924+0000 7fc4dfbfd700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d819d570 con 0x7fc4d80ffd10 2023-07-14T17:44:48.909 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.924+0000 7fc4dfbfd700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d819d730 con 0x7fc4d81006a0 2023-07-14T17:44:48.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.924+0000 7fc4dd999700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4d80feb20 0x7fc4d8069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:48.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.924+0000 7fc4dd999700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4d80feb20 0x7fc4d8069340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43726/0 (socket says 172.21.15.88:43726) 2023-07-14T17:44:48.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.924+0000 7fc4dd999700 1 -- 172.21.15.88:0/3734291003 learned_addr learned my addr 172.21.15.88:0/3734291003 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:48.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.924+0000 7fc4de19a700 1 --2- 172.21.15.88:0/3734291003 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc4d81006a0 0x7fc4d819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:48.910 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.924+0000 7fc4dd198700 1 --2- 172.21.15.88:0/3734291003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4d80ffd10 0x7fc4d819a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:48.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.924+0000 7fc4dd999700 1 -- 172.21.15.88:0/3734291003 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc4d81006a0 msgr2=0x7fc4d819cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:48.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.924+0000 7fc4dd999700 1 --2- 172.21.15.88:0/3734291003 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc4d81006a0 0x7fc4d819cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:48.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.924+0000 7fc4dd999700 1 -- 172.21.15.88:0/3734291003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4d80ffd10 msgr2=0x7fc4d819a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:48.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.924+0000 7fc4dd999700 1 --2- 172.21.15.88:0/3734291003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4d80ffd10 0x7fc4d819a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:48.911 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.924+0000 7fc4dd999700 1 -- 172.21.15.88:0/3734291003 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d40095f0 con 0x7fc4d80feb20 2023-07-14T17:44:48.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.924+0000 7fc4dd999700 1 --2- 172.21.15.88:0/3734291003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4d80feb20 0x7fc4d8069340 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4000c00 tx=0x7fc4d40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:48.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.925+0000 7fc4ceffd700 1 -- 172.21.15.88:0/3734291003 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d401d070 con 0x7fc4d80feb20 2023-07-14T17:44:48.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.925+0000 7fc4dfbfd700 1 -- 172.21.15.88:0/3734291003 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d819d9b0 con 0x7fc4d80feb20 2023-07-14T17:44:48.912 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.925+0000 7fc4ceffd700 1 -- 172.21.15.88:0/3734291003 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc4d4003f40 con 0x7fc4d80feb20 2023-07-14T17:44:48.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.925+0000 7fc4ceffd700 1 -- 172.21.15.88:0/3734291003 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d4022bf0 con 0x7fc4d80feb20 2023-07-14T17:44:48.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.925+0000 7fc4dfbfd700 1 -- 172.21.15.88:0/3734291003 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d819de60 con 0x7fc4d80feb20 2023-07-14T17:44:48.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.926+0000 7fc4dfbfd700 1 -- 172.21.15.88:0/3734291003 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d81060b0 con 0x7fc4d80feb20 2023-07-14T17:44:48.913 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.927+0000 7fc4ceffd700 1 -- 172.21.15.88:0/3734291003 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc4d4022d50 con 0x7fc4d80feb20 2023-07-14T17:44:48.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.927+0000 7fc4ceffd700 1 --2- 172.21.15.88:0/3734291003 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc4c40a11e0 0x7fc4c40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:48.914 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.928+0000 7fc4dd198700 1 --2- 172.21.15.88:0/3734291003 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc4c40a11e0 0x7fc4c40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:48.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.928+0000 7fc4ceffd700 1 -- 172.21.15.88:0/3734291003 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9508+0+0 (secure 0 0 0) 0x7fc4d40ce880 con 0x7fc4d80feb20 2023-07-14T17:44:48.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.928+0000 7fc4dd198700 1 --2- 172.21.15.88:0/3734291003 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc4c40a11e0 0x7fc4c40a3690 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80690b0 tx=0x7fc4c800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:48.915 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:48.931+0000 7fc4ceffd700 1 -- 172.21.15.88:0/3734291003 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc4d4097a00 con 0x7fc4d80feb20 2023-07-14T17:44:49.080 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: pgmap v323: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 511 B/s wr, 10 op/s 2023-07-14T17:44:49.081 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/239019189' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T17:44:49.081 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:44:49.081 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T17:44:49.082 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T17:44:49.082 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: Cluster is now healthy 2023-07-14T17:44:49.082 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.107:6833/948534928,v1:172.21.15.107:6837/948534928] up:boot 2023-07-14T17:44:49.082 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.178:6834/381057656,v1:172.21.15.178:6835/381057656] up:boot 2023-07-14T17:44:49.083 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.88:6838/4165981347,v1:172.21.15.88:6841/4165981347] up:boot 2023-07-14T17:44:49.083 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.88:6839/3472342514,v1:172.21.15.88:6840/3472342514] up:boot 2023-07-14T17:44:49.083 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.107:6838/3907351061,v1:172.21.15.107:6839/3907351061] up:boot 2023-07-14T17:44:49.083 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.178:6836/2004388235,v1:172.21.15.178:6838/2004388235] up:boot 2023-07-14T17:44:49.084 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.178:6837/1346558042,v1:172.21.15.178:6839/1346558042] up:boot 2023-07-14T17:44:49.084 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.107:6832/359092006,v1:172.21.15.107:6835/359092006] up:boot 2023-07-14T17:44:49.084 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.107:6834/819366129,v1:172.21.15.107:6836/819366129] up:boot 2023-07-14T17:44:49.085 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: mds.? [v2:172.21.15.178:6832/4147680704,v1:172.21.15.178:6833/4147680704] up:boot 2023-07-14T17:44:49.085 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/239019189' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T17:44:49.085 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: fsmap 12 up:standby 2023-07-14T17:44:49.086 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T17:44:49.086 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T17:44:49.086 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T17:44:49.086 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T17:44:49.086 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T17:44:49.087 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T17:44:49.087 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T17:44:49.087 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T17:44:49.087 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T17:44:49.088 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:49 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T17:44:49.114 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:49.132+0000 7fc4dfbfd700 1 -- 172.21.15.88:0/3734291003 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc4d802c8e0 con 0x7fc4d80feb20 2023-07-14T17:44:49.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: pgmap v323: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 511 B/s wr, 10 op/s 2023-07-14T17:44:49.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/239019189' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T17:44:49.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:44:49.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T17:44:49.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T17:44:49.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: Cluster is now healthy 2023-07-14T17:44:49.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.107:6833/948534928,v1:172.21.15.107:6837/948534928] up:boot 2023-07-14T17:44:49.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.178:6834/381057656,v1:172.21.15.178:6835/381057656] up:boot 2023-07-14T17:44:49.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.88:6838/4165981347,v1:172.21.15.88:6841/4165981347] up:boot 2023-07-14T17:44:49.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.88:6839/3472342514,v1:172.21.15.88:6840/3472342514] up:boot 2023-07-14T17:44:49.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.107:6838/3907351061,v1:172.21.15.107:6839/3907351061] up:boot 2023-07-14T17:44:49.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.178:6836/2004388235,v1:172.21.15.178:6838/2004388235] up:boot 2023-07-14T17:44:49.416 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.178:6837/1346558042,v1:172.21.15.178:6839/1346558042] up:boot 2023-07-14T17:44:49.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.107:6832/359092006,v1:172.21.15.107:6835/359092006] up:boot 2023-07-14T17:44:49.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.107:6834/819366129,v1:172.21.15.107:6836/819366129] up:boot 2023-07-14T17:44:49.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: mds.? [v2:172.21.15.178:6832/4147680704,v1:172.21.15.178:6833/4147680704] up:boot 2023-07-14T17:44:49.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/239019189' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T17:44:49.417 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: fsmap 12 up:standby 2023-07-14T17:44:49.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T17:44:49.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T17:44:49.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T17:44:49.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T17:44:49.418 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T17:44:49.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T17:44:49.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T17:44:49.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T17:44:49.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T17:44:49.419 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:49 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T17:44:49.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: pgmap v323: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 511 B/s wr, 10 op/s 2023-07-14T17:44:49.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/239019189' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T17:44:49.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T17:44:49.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T17:44:49.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T17:44:49.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: Cluster is now healthy 2023-07-14T17:44:49.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.107:6833/948534928,v1:172.21.15.107:6837/948534928] up:boot 2023-07-14T17:44:49.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.178:6834/381057656,v1:172.21.15.178:6835/381057656] up:boot 2023-07-14T17:44:49.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.88:6838/4165981347,v1:172.21.15.88:6841/4165981347] up:boot 2023-07-14T17:44:49.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.88:6839/3472342514,v1:172.21.15.88:6840/3472342514] up:boot 2023-07-14T17:44:49.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.107:6838/3907351061,v1:172.21.15.107:6839/3907351061] up:boot 2023-07-14T17:44:49.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.178:6836/2004388235,v1:172.21.15.178:6838/2004388235] up:boot 2023-07-14T17:44:49.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.178:6837/1346558042,v1:172.21.15.178:6839/1346558042] up:boot 2023-07-14T17:44:49.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.107:6832/359092006,v1:172.21.15.107:6835/359092006] up:boot 2023-07-14T17:44:49.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.107:6834/819366129,v1:172.21.15.107:6836/819366129] up:boot 2023-07-14T17:44:49.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: mds.? [v2:172.21.15.178:6832/4147680704,v1:172.21.15.178:6833/4147680704] up:boot 2023-07-14T17:44:49.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/239019189' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T17:44:49.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: fsmap 12 up:standby 2023-07-14T17:44:49.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T17:44:49.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T17:44:49.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T17:44:49.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T17:44:49.478 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T17:44:49.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T17:44:49.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T17:44:49.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T17:44:49.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T17:44:49.479 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:49 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T17:44:50.092 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.111+0000 7fc4ceffd700 1 -- 172.21.15.88:0/3734291003 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v92) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc4d409b020 con 0x7fc4d80feb20 2023-07-14T17:44:50.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.115+0000 7fc4dfbfd700 1 -- 172.21.15.88:0/3734291003 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc4c40a11e0 msgr2=0x7fc4c40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:50.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.115+0000 7fc4dfbfd700 1 --2- 172.21.15.88:0/3734291003 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc4c40a11e0 0x7fc4c40a3690 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc4d80690b0 tx=0x7fc4c800b410 comp rx=0 tx=0).stop 2023-07-14T17:44:50.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.115+0000 7fc4dfbfd700 1 -- 172.21.15.88:0/3734291003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4d80feb20 msgr2=0x7fc4d8069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:50.097 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.115+0000 7fc4dfbfd700 1 --2- 172.21.15.88:0/3734291003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4d80feb20 0x7fc4d8069340 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4000c00 tx=0x7fc4d40047b0 comp rx=0 tx=0).stop 2023-07-14T17:44:50.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.116+0000 7fc4dfbfd700 1 -- 172.21.15.88:0/3734291003 shutdown_connections 2023-07-14T17:44:50.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.116+0000 7fc4dfbfd700 1 --2- 172.21.15.88:0/3734291003 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fc4d81006a0 0x7fc4d819cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:50.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.116+0000 7fc4dfbfd700 1 --2- 172.21.15.88:0/3734291003 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fc4c40a11e0 0x7fc4c40a3690 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:50.098 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.116+0000 7fc4dfbfd700 1 --2- 172.21.15.88:0/3734291003 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc4d80feb20 0x7fc4d8069340 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:50.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.116+0000 7fc4dfbfd700 1 --2- 172.21.15.88:0/3734291003 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fc4d80ffd10 0x7fc4d819a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:50.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.116+0000 7fc4dfbfd700 1 -- 172.21.15.88:0/3734291003 >> 172.21.15.88:0/3734291003 conn(0x7fc4d80fa160 msgr2=0x7fc4d81031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:50.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.116+0000 7fc4dfbfd700 1 -- 172.21.15.88:0/3734291003 shutdown_connections 2023-07-14T17:44:50.099 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.116+0000 7fc4dfbfd700 1 -- 172.21.15.88:0/3734291003 wait complete. 2023-07-14T17:44:50.100 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_metadata' removed 2023-07-14T17:44:50.111 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-07-14T17:44:50.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.259+0000 7f50ebba0700 1 -- 172.21.15.88:0/78841920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50e41018c0 msgr2=0x7f50e4108180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:50.243 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.259+0000 7f50ebba0700 1 --2- 172.21.15.88:0/78841920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50e41018c0 0x7f50e4108180 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f50dc00b0a0 tx=0x7f50dc00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:50.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.259+0000 7f50ebba0700 1 -- 172.21.15.88:0/78841920 shutdown_connections 2023-07-14T17:44:50.244 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.259+0000 7f50ebba0700 1 --2- 172.21.15.88:0/78841920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50e41018c0 0x7f50e4108180 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:50.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.259+0000 7f50ebba0700 1 --2- 172.21.15.88:0/78841920 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e40fdec0 0x7f50e40fe2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:50.245 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.259+0000 7f50ebba0700 1 --2- 172.21.15.88:0/78841920 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50e4100f30 0x7f50e4101380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:50.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.259+0000 7f50ebba0700 1 -- 172.21.15.88:0/78841920 >> 172.21.15.88:0/78841920 conn(0x7f50e40f98a0 msgr2=0x7f50e40fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:50.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.259+0000 7f50ebba0700 1 -- 172.21.15.88:0/78841920 shutdown_connections 2023-07-14T17:44:50.246 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.259+0000 7f50ebba0700 1 -- 172.21.15.88:0/78841920 wait complete. 2023-07-14T17:44:50.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.259+0000 7f50ebba0700 1 Processor -- start 2023-07-14T17:44:50.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.259+0000 7f50ebba0700 1 -- start start 2023-07-14T17:44:50.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50ebba0700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50e40fdec0 0x7f50e41008a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:50.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50ebba0700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e4100f30 0x7f50e40feee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:50.247 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50ebba0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50e41018c0 0x7f50e40ff420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:50.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50ebba0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e40ffbc0 con 0x7f50e41018c0 2023-07-14T17:44:50.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50ebba0700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e40ffd30 con 0x7f50e40fdec0 2023-07-14T17:44:50.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50ebba0700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e419fa40 con 0x7f50e4100f30 2023-07-14T17:44:50.248 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50ea13d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50e41018c0 0x7f50e40ff420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:50.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50e913b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e4100f30 0x7f50e40feee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:50.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50e913b700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e4100f30 0x7f50e40feee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.178:3300/0 says I am v2:172.21.15.88:33274/0 (socket says 172.21.15.88:33274) 2023-07-14T17:44:50.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50e993c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50e40fdec0 0x7f50e41008a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:50.249 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50e913b700 1 -- 172.21.15.88:0/2698169781 learned_addr learned my addr 172.21.15.88:0/2698169781 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:50.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50e913b700 1 -- 172.21.15.88:0/2698169781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50e40fdec0 msgr2=0x7f50e41008a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:50.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50e913b700 1 --2- 172.21.15.88:0/2698169781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50e40fdec0 0x7f50e41008a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:50.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50e913b700 1 -- 172.21.15.88:0/2698169781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50e41018c0 msgr2=0x7f50e40ff420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:50.250 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50e913b700 1 --2- 172.21.15.88:0/2698169781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50e41018c0 0x7f50e40ff420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:50.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50e913b700 1 -- 172.21.15.88:0/2698169781 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50dc009d00 con 0x7f50e4100f30 2023-07-14T17:44:50.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50e993c700 1 --2- 172.21.15.88:0/2698169781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50e40fdec0 0x7f50e41008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T17:44:50.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50ea13d700 1 --2- 172.21.15.88:0/2698169781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50e41018c0 0x7f50e40ff420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T17:44:50.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50e913b700 1 --2- 172.21.15.88:0/2698169781 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e4100f30 0x7f50e40feee0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f50d400e9a0 tx=0x7f50d400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:50.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.260+0000 7f50daffd700 1 -- 172.21.15.88:0/2698169781 <== mon.1 v2:172.21.15.178:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d4009790 con 0x7f50e4100f30 2023-07-14T17:44:50.251 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.261+0000 7f50ebba0700 1 -- 172.21.15.88:0/2698169781 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e419fd20 con 0x7f50e4100f30 2023-07-14T17:44:50.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.261+0000 7f50daffd700 1 -- 172.21.15.88:0/2698169781 <== mon.1 v2:172.21.15.178:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f50d4010430 con 0x7f50e4100f30 2023-07-14T17:44:50.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.261+0000 7f50ebba0700 1 -- 172.21.15.88:0/2698169781 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e41a01f0 con 0x7f50e4100f30 2023-07-14T17:44:50.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.261+0000 7f50daffd700 1 -- 172.21.15.88:0/2698169781 <== mon.1 v2:172.21.15.178:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d400cea0 con 0x7f50e4100f30 2023-07-14T17:44:50.252 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.262+0000 7f50ebba0700 1 -- 172.21.15.88:0/2698169781 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e410b2e0 con 0x7f50e4100f30 2023-07-14T17:44:50.253 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.263+0000 7f50daffd700 1 -- 172.21.15.88:0/2698169781 <== mon.1 v2:172.21.15.178:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f50d40105a0 con 0x7f50e4100f30 2023-07-14T17:44:50.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.263+0000 7f50daffd700 1 --2- 172.21.15.88:0/2698169781 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f50d00a11e0 0x7f50d00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:50.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.263+0000 7f50daffd700 1 -- 172.21.15.88:0/2698169781 <== mon.1 v2:172.21.15.178:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8983+0+0 (secure 0 0 0) 0x7f50d4014070 con 0x7f50e4100f30 2023-07-14T17:44:50.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.263+0000 7f50e993c700 1 --2- 172.21.15.88:0/2698169781 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f50d00a11e0 0x7f50d00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:50.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.264+0000 7f50e993c700 1 --2- 172.21.15.88:0/2698169781 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f50d00a11e0 0x7f50d00a3690 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f50e4067500 tx=0x7f50e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:50.254 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.268+0000 7f50daffd700 1 -- 172.21.15.88:0/2698169781 <== mon.1 v2:172.21.15.178:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f50d4051780 con 0x7f50e4100f30 2023-07-14T17:44:50.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:50 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3734291003' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T17:44:50.440 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:50 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3734291003' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T17:44:50.454 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:50.473+0000 7f50ebba0700 1 -- 172.21.15.88:0/2698169781 --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f50e410b4a0 con 0x7f50e4100f30 2023-07-14T17:44:50.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:50 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3734291003' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T17:44:51.096 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.115+0000 7f50daffd700 1 -- 172.21.15.88:0/2698169781 <== mon.1 v2:172.21.15.178:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v93) v1 ==== 169+0+0 (secure 0 0 0) 0x7f50d4096280 con 0x7f50e4100f30 2023-07-14T17:44:51.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.118+0000 7f50ebba0700 1 -- 172.21.15.88:0/2698169781 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f50d00a11e0 msgr2=0x7f50d00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:51.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.118+0000 7f50ebba0700 1 --2- 172.21.15.88:0/2698169781 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f50d00a11e0 0x7f50d00a3690 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f50e4067500 tx=0x7f50e0009260 comp rx=0 tx=0).stop 2023-07-14T17:44:51.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.119+0000 7f50ebba0700 1 -- 172.21.15.88:0/2698169781 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e4100f30 msgr2=0x7f50e40feee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:51.100 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.119+0000 7f50ebba0700 1 --2- 172.21.15.88:0/2698169781 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e4100f30 0x7f50e40feee0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f50d400e9a0 tx=0x7f50d400ecb0 comp rx=0 tx=0).stop 2023-07-14T17:44:51.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.119+0000 7f50ebba0700 1 -- 172.21.15.88:0/2698169781 shutdown_connections 2023-07-14T17:44:51.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.119+0000 7f50ebba0700 1 --2- 172.21.15.88:0/2698169781 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7f50d00a11e0 0x7f50d00a3690 secure :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f50e4067500 tx=0x7f50e0009260 comp rx=0 tx=0).stop 2023-07-14T17:44:51.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.119+0000 7f50ebba0700 1 --2- 172.21.15.88:0/2698169781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50e41018c0 0x7f50e40ff420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:51.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.119+0000 7f50ebba0700 1 --2- 172.21.15.88:0/2698169781 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7f50e40fdec0 0x7f50e41008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:51.101 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.119+0000 7f50ebba0700 1 --2- 172.21.15.88:0/2698169781 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7f50e4100f30 0x7f50e40feee0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:51.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.119+0000 7f50ebba0700 1 -- 172.21.15.88:0/2698169781 >> 172.21.15.88:0/2698169781 conn(0x7f50e40f98a0 msgr2=0x7f50e40fbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:51.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.119+0000 7f50ebba0700 1 -- 172.21.15.88:0/2698169781 shutdown_connections 2023-07-14T17:44:51.102 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.119+0000 7f50ebba0700 1 -- 172.21.15.88:0/2698169781 wait complete. 2023-07-14T17:44:51.103 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data' removed 2023-07-14T17:44:51.113 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-07-14T17:44:51.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.285+0000 7fb608cb9700 1 -- 172.21.15.88:0/3795044562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb604105900 msgr2=0x7fb604107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:51.266 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.285+0000 7fb608cb9700 1 --2- 172.21.15.88:0/3795044562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb604105900 0x7fb604107d60 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00b0a0 tx=0x7fb5fc00b3b0 comp rx=0 tx=0).stop 2023-07-14T17:44:51.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.285+0000 7fb608cb9700 1 -- 172.21.15.88:0/3795044562 shutdown_connections 2023-07-14T17:44:51.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.285+0000 7fb608cb9700 1 --2- 172.21.15.88:0/3795044562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb604105900 0x7fb604107d60 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:51.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.285+0000 7fb608cb9700 1 --2- 172.21.15.88:0/3795044562 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb6041006c0 0x7fb604102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:51.267 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.285+0000 7fb608cb9700 1 --2- 172.21.15.88:0/3795044562 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb604102fe0 0x7fb6041053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:51.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.285+0000 7fb608cb9700 1 -- 172.21.15.88:0/3795044562 >> 172.21.15.88:0/3795044562 conn(0x7fb6040fa160 msgr2=0x7fb6040fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:51.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.285+0000 7fb608cb9700 1 -- 172.21.15.88:0/3795044562 shutdown_connections 2023-07-14T17:44:51.268 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.285+0000 7fb608cb9700 1 -- 172.21.15.88:0/3795044562 wait complete. 2023-07-14T17:44:51.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.286+0000 7fb608cb9700 1 Processor -- start 2023-07-14T17:44:51.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.286+0000 7fb608cb9700 1 -- start start 2023-07-14T17:44:51.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.286+0000 7fb608cb9700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6041006c0 0x7fb604195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:51.270 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.286+0000 7fb608cb9700 1 --2- >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb604102fe0 0x7fb604195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:51.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.286+0000 7fb608cb9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb604105900 0x7fb60419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:51.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.286+0000 7fb608cb9700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60419d4b0 con 0x7fb604105900 2023-07-14T17:44:51.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.286+0000 7fb608cb9700 1 -- --> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60419d5f0 con 0x7fb6041006c0 2023-07-14T17:44:51.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.286+0000 7fb608cb9700 1 -- --> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60419d780 con 0x7fb604102fe0 2023-07-14T17:44:51.271 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.286+0000 7fb602d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb604105900 0x7fb60419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:51.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.286+0000 7fb602d9d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb604105900 0x7fb60419cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:43756/0 (socket says 172.21.15.88:43756) 2023-07-14T17:44:51.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.286+0000 7fb60259c700 1 --2- >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6041006c0 0x7fb604195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:51.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.286+0000 7fb602d9d700 1 -- 172.21.15.88:0/3338293498 learned_addr learned my addr 172.21.15.88:0/3338293498 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T17:44:51.272 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.286+0000 7fb601d9b700 1 --2- 172.21.15.88:0/3338293498 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb604102fe0 0x7fb604195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:51.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.287+0000 7fb602d9d700 1 -- 172.21.15.88:0/3338293498 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb604102fe0 msgr2=0x7fb604195e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:51.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.287+0000 7fb602d9d700 1 --2- 172.21.15.88:0/3338293498 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb604102fe0 0x7fb604195e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:51.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.287+0000 7fb602d9d700 1 -- 172.21.15.88:0/3338293498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6041006c0 msgr2=0x7fb604195930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:51.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.287+0000 7fb602d9d700 1 --2- 172.21.15.88:0/3338293498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6041006c0 0x7fb604195930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:51.273 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.287+0000 7fb602d9d700 1 -- 172.21.15.88:0/3338293498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc009d00 con 0x7fb604105900 2023-07-14T17:44:51.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.287+0000 7fb60259c700 1 --2- 172.21.15.88:0/3338293498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6041006c0 0x7fb604195930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T17:44:51.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.287+0000 7fb602d9d700 1 --2- 172.21.15.88:0/3338293498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb604105900 0x7fb60419cee0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00b780 tx=0x7fb5fc0078e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:51.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.287+0000 7fb5f37fe700 1 -- 172.21.15.88:0/3338293498 <== mon.0 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc00e040 con 0x7fb604105900 2023-07-14T17:44:51.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.287+0000 7fb5f37fe700 1 -- 172.21.15.88:0/3338293498 <== mon.0 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb5fc00be80 con 0x7fb604105900 2023-07-14T17:44:51.274 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.287+0000 7fb5f37fe700 1 -- 172.21.15.88:0/3338293498 <== mon.0 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc01dea0 con 0x7fb604105900 2023-07-14T17:44:51.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.287+0000 7fb608cb9700 1 -- 172.21.15.88:0/3338293498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60419da00 con 0x7fb604105900 2023-07-14T17:44:51.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.287+0000 7fb608cb9700 1 -- 172.21.15.88:0/3338293498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60419de30 con 0x7fb604105900 2023-07-14T17:44:51.275 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.288+0000 7fb608cb9700 1 -- 172.21.15.88:0/3338293498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6040fbcd0 con 0x7fb604105900 2023-07-14T17:44:51.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.289+0000 7fb5f37fe700 1 -- 172.21.15.88:0/3338293498 <== mon.0 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb5fc01d470 con 0x7fb604105900 2023-07-14T17:44:51.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.290+0000 7fb5f37fe700 1 --2- 172.21.15.88:0/3338293498 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb5ec0a1260 0x7fb5ec0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T17:44:51.276 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.290+0000 7fb60259c700 1 --2- 172.21.15.88:0/3338293498 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb5ec0a1260 0x7fb5ec0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T17:44:51.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.290+0000 7fb5f37fe700 1 -- 172.21.15.88:0/3338293498 <== mon.0 v2:172.21.15.88:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8424+0+0 (secure 0 0 0) 0x7fb5fc0ce380 con 0x7fb604105900 2023-07-14T17:44:51.277 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.290+0000 7fb60259c700 1 --2- 172.21.15.88:0/3338293498 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb5ec0a1260 0x7fb5ec0a3710 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4011490 tx=0x7fb5f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T17:44:51.278 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.293+0000 7fb5f37fe700 1 -- 172.21.15.88:0/3338293498 <== mon.0 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb5fc0978e0 con 0x7fb604105900 2023-07-14T17:44:51.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:51 smithi178 ceph-mon[129368]: pgmap v324: 129 pgs: 129 active+clean; 650 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2023-07-14T17:44:51.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:51 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3734291003' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T17:44:51.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:51 smithi178 ceph-mon[129368]: osdmap e92: 12 total, 12 up, 12 in 2023-07-14T17:44:51.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:51 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/2698169781' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T17:44:51.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:51 smithi178 ceph-mon[129368]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T17:44:51.440 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:51 smithi088 ceph-mon[127465]: pgmap v324: 129 pgs: 129 active+clean; 650 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2023-07-14T17:44:51.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:51 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3734291003' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T17:44:51.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:51 smithi088 ceph-mon[127465]: osdmap e92: 12 total, 12 up, 12 in 2023-07-14T17:44:51.441 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:51 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/2698169781' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T17:44:51.442 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:51 smithi088 ceph-mon[127465]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T17:44:51.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:51 smithi107 ceph-mon[128999]: pgmap v324: 129 pgs: 129 active+clean; 650 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2023-07-14T17:44:51.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:51 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3734291003' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T17:44:51.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:51 smithi107 ceph-mon[128999]: osdmap e92: 12 total, 12 up, 12 in 2023-07-14T17:44:51.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:51 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/2698169781' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T17:44:51.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:51 smithi107 ceph-mon[128999]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T17:44:51.487 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:51.506+0000 7fb608cb9700 1 -- 172.21.15.88:0/3338293498 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb60402c8e0 con 0x7fb604105900 2023-07-14T17:44:52.187 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:52.206+0000 7fb5f37fe700 1 -- 172.21.15.88:0/3338293498 <== mon.0 v2:172.21.15.88:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v94) v1 ==== 178+0+0 (secure 0 0 0) 0x7fb5fc004c50 con 0x7fb604105900 2023-07-14T17:44:52.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:52.210+0000 7fb608cb9700 1 -- 172.21.15.88:0/3338293498 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb5ec0a1260 msgr2=0x7fb5ec0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:52.191 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:52.210+0000 7fb608cb9700 1 --2- 172.21.15.88:0/3338293498 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb5ec0a1260 0x7fb5ec0a3710 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4011490 tx=0x7fb5f4009260 comp rx=0 tx=0).stop 2023-07-14T17:44:52.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:52.210+0000 7fb608cb9700 1 -- 172.21.15.88:0/3338293498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb604105900 msgr2=0x7fb60419cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T17:44:52.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:52.210+0000 7fb608cb9700 1 --2- 172.21.15.88:0/3338293498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb604105900 0x7fb60419cee0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc00b780 tx=0x7fb5fc0078e0 comp rx=0 tx=0).stop 2023-07-14T17:44:52.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:52.210+0000 7fb608cb9700 1 -- 172.21.15.88:0/3338293498 shutdown_connections 2023-07-14T17:44:52.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:52.210+0000 7fb608cb9700 1 --2- 172.21.15.88:0/3338293498 >> [v2:172.21.15.88:6800/4262759610,v1:172.21.15.88:6801/4262759610] conn(0x7fb5ec0a1260 0x7fb5ec0a3710 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:52.192 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:52.210+0000 7fb608cb9700 1 --2- 172.21.15.88:0/3338293498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb604105900 0x7fb60419cee0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:52.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:52.210+0000 7fb608cb9700 1 --2- 172.21.15.88:0/3338293498 >> [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] conn(0x7fb6041006c0 0x7fb604195930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:52.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:52.210+0000 7fb608cb9700 1 --2- 172.21.15.88:0/3338293498 >> [v2:172.21.15.178:3300/0,v1:172.21.15.178:6789/0] conn(0x7fb604102fe0 0x7fb604195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T17:44:52.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:52.210+0000 7fb608cb9700 1 -- 172.21.15.88:0/3338293498 >> 172.21.15.88:0/3338293498 conn(0x7fb6040fa160 msgr2=0x7fb604101320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T17:44:52.193 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:52.210+0000 7fb608cb9700 1 -- 172.21.15.88:0/3338293498 shutdown_connections 2023-07-14T17:44:52.194 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T17:44:52.210+0000 7fb608cb9700 1 -- 172.21.15.88:0/3338293498 wait complete. 2023-07-14T17:44:52.195 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'cephfs_data_ec' removed 2023-07-14T17:44:52.205 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T17:44:52.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T17:44:52.278 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T17:44:52.322 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-14T17:44:52.322 DEBUG:teuthology.orchestra.run.smithi088:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T17:44:52.340 DEBUG:teuthology.orchestra.run.smithi107:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T17:44:52.357 DEBUG:teuthology.orchestra.run.smithi178:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T17:44:52.379 INFO:tasks.cephadm:Stopping all daemons... 2023-07-14T17:44:52.380 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-14T17:44:52.380 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@mon.a 2023-07-14T17:44:52.404 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:52 smithi088 ceph-mon[127465]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T17:44:52.404 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:52 smithi088 ceph-mon[127465]: osdmap e93: 12 total, 12 up, 12 in 2023-07-14T17:44:52.404 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:52 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.405 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:52 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.405 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:52 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.405 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:52 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.405 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:52 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.405 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:52 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.406 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:52 smithi088 ceph-mon[127465]: from='client.? 172.21.15.88:0/3338293498' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T17:44:52.406 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:52 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:44:52.406 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:52 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:44:52.406 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:52 smithi088 ceph-mon[127465]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:52 smithi178 ceph-mon[129368]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T17:44:52.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:52 smithi178 ceph-mon[129368]: osdmap e93: 12 total, 12 up, 12 in 2023-07-14T17:44:52.413 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:52 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:52 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:52 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:52 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:52 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.414 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:52 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:52 smithi178 ceph-mon[129368]: from='client.? 172.21.15.88:0/3338293498' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T17:44:52.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:52 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:44:52.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:52 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:44:52.415 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:52 smithi178 ceph-mon[129368]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:52 smithi107 ceph-mon[128999]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T17:44:52.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:52 smithi107 ceph-mon[128999]: osdmap e93: 12 total, 12 up, 12 in 2023-07-14T17:44:52.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:52 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:52 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:52 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.474 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:52 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:52 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:52 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:52 smithi107 ceph-mon[128999]: from='client.? 172.21.15.88:0/3338293498' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T17:44:52.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:52 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T17:44:52.475 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:52 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T17:44:52.476 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:52 smithi107 ceph-mon[128999]: from='mgr.14152 172.21.15.88:0/1899726560' entity='mgr.x' 2023-07-14T17:44:52.691 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:52 smithi088 systemd[1]: Stopping Ceph mon.a for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:44:53.003 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:52 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mon-a[127441]: 2023-07-14T17:44:52.749+0000 7f904c1ec700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:44:53.003 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:52 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mon-a[127441]: 2023-07-14T17:44:52.749+0000 7f904c1ec700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-14T17:44:53.256 INFO:journalctl@ceph.mon.a.smithi088.stdout:Jul 14 17:44:53 smithi088 bash[180534]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-mon-a 2023-07-14T17:44:53.263 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mon.a.service' 2023-07-14T17:44:53.350 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:44:53.351 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-14T17:44:53.351 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-07-14T17:44:53.352 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@mon.b 2023-07-14T17:44:53.724 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:53 smithi107 systemd[1]: Stopping Ceph mon.b for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:44:54.067 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:53 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mon-b[128975]: 2023-07-14T17:44:53.742+0000 7eff46f9f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:44:54.068 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:53 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mon-b[128975]: 2023-07-14T17:44:53.742+0000 7eff46f9f700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-07-14T17:44:54.473 INFO:journalctl@ceph.mon.b.smithi107.stdout:Jul 14 17:44:54 smithi107 bash[156207]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-mon-b 2023-07-14T17:44:54.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mon.b.service' 2023-07-14T17:44:54.629 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:44:54.630 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-07-14T17:44:54.630 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-07-14T17:44:54.630 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@mon.c 2023-07-14T17:44:54.913 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:54 smithi178 systemd[1]: Stopping Ceph mon.c for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:44:55.298 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:55 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mon-c[129322]: 2023-07-14T17:44:55.024+0000 7f38939e3700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:44:55.298 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:55 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mon-c[129322]: 2023-07-14T17:44:55.024+0000 7f38939e3700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-07-14T17:44:55.557 INFO:journalctl@ceph.mon.c.smithi178.stdout:Jul 14 17:44:55 smithi178 bash[157719]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-mon-c 2023-07-14T17:44:55.745 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mon.c.service' 2023-07-14T17:44:55.805 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:44:55.806 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-07-14T17:44:55.806 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-14T17:44:55.807 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@mgr.x 2023-07-14T17:44:56.155 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:44:55 smithi088 systemd[1]: Stopping Ceph mgr.x for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:44:56.436 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:44:56 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:44:56.173+0000 7f4522056700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:44:56.436 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:44:56 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x[127710]: 2023-07-14T17:44:56.173+0000 7f4522056700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-07-14T17:44:56.690 INFO:journalctl@ceph.mgr.x.smithi088.stdout:Jul 14 17:44:56 smithi088 bash[180673]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-x 2023-07-14T17:44:56.699 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mgr.x.service' 2023-07-14T17:44:56.738 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:44:56.739 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-14T17:44:56.739 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-14T17:44:56.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@mgr.y 2023-07-14T17:44:57.145 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:44:56 smithi107 systemd[1]: Stopping Ceph mgr.y for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:44:57.442 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:44:57 smithi107 bash[156368]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-y 2023-07-14T17:44:57.626 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Jul 14 17:44:57 smithi107 systemd[1]: ceph-b462a556-226c-11ee-9b33-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-07-14T17:44:57.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mgr.y.service' 2023-07-14T17:44:57.665 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:44:57.666 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-14T17:44:57.666 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-07-14T17:44:57.667 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@mgr.z 2023-07-14T17:44:58.086 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:44:57 smithi178 systemd[1]: Stopping Ceph mgr.z for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:44:58.413 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:44:58 smithi178 bash[157873]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-mgr-z 2023-07-14T17:44:58.414 INFO:journalctl@ceph.mgr.z.smithi178.stdout:Jul 14 17:44:58 smithi178 systemd[1]: ceph-b462a556-226c-11ee-9b33-001a4aab830c@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-07-14T17:44:58.460 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mgr.z.service' 2023-07-14T17:44:58.498 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:44:58.498 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-07-14T17:44:58.499 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-14T17:44:58.500 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.0 2023-07-14T17:44:58.862 INFO:journalctl@ceph.osd.0.smithi088.stdout:Jul 14 17:44:58 smithi088 systemd[1]: Stopping Ceph osd.0 for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:44:59.190 INFO:journalctl@ceph.osd.0.smithi088.stdout:Jul 14 17:44:58 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-0[140436]: 2023-07-14T17:44:58.879+0000 7fe364e85700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:44:59.191 INFO:journalctl@ceph.osd.0.smithi088.stdout:Jul 14 17:44:58 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-0[140436]: 2023-07-14T17:44:58.879+0000 7fe364e85700 -1 osd.0 94 *** Got signal Terminated *** 2023-07-14T17:44:59.191 INFO:journalctl@ceph.osd.0.smithi088.stdout:Jul 14 17:44:58 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-0[140436]: 2023-07-14T17:44:58.879+0000 7fe364e85700 -1 osd.0 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T17:45:04.631 INFO:journalctl@ceph.osd.0.smithi088.stdout:Jul 14 17:45:04 smithi088 bash[180811]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-0 2023-07-14T17:45:05.896 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.0.service' 2023-07-14T17:45:05.937 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:45:05.938 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-14T17:45:05.938 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-14T17:45:05.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.1 2023-07-14T17:45:06.223 INFO:journalctl@ceph.osd.1.smithi107.stdout:Jul 14 17:45:05 smithi107 systemd[1]: Stopping Ceph osd.1 for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:45:06.723 INFO:journalctl@ceph.osd.1.smithi107.stdout:Jul 14 17:45:06 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-1[133955]: 2023-07-14T17:45:06.324+0000 7f571512c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:45:06.723 INFO:journalctl@ceph.osd.1.smithi107.stdout:Jul 14 17:45:06 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-1[133955]: 2023-07-14T17:45:06.324+0000 7f571512c700 -1 osd.1 94 *** Got signal Terminated *** 2023-07-14T17:45:06.724 INFO:journalctl@ceph.osd.1.smithi107.stdout:Jul 14 17:45:06 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-1[133955]: 2023-07-14T17:45:06.324+0000 7f571512c700 -1 osd.1 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T17:45:11.973 INFO:journalctl@ceph.osd.1.smithi107.stdout:Jul 14 17:45:11 smithi107 bash[156508]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-1 2023-07-14T17:45:13.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.1.service' 2023-07-14T17:45:13.292 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:45:13.293 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-14T17:45:13.293 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-14T17:45:13.294 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.2 2023-07-14T17:45:13.644 INFO:journalctl@ceph.osd.2.smithi178.stdout:Jul 14 17:45:13 smithi178 systemd[1]: Stopping Ceph osd.2 for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:45:13.913 INFO:journalctl@ceph.osd.2.smithi178.stdout:Jul 14 17:45:13 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-2[135188]: 2023-07-14T17:45:13.662+0000 7fd1c4807700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:45:13.913 INFO:journalctl@ceph.osd.2.smithi178.stdout:Jul 14 17:45:13 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-2[135188]: 2023-07-14T17:45:13.662+0000 7fd1c4807700 -1 osd.2 94 *** Got signal Terminated *** 2023-07-14T17:45:13.913 INFO:journalctl@ceph.osd.2.smithi178.stdout:Jul 14 17:45:13 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-2[135188]: 2023-07-14T17:45:13.662+0000 7fd1c4807700 -1 osd.2 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T17:45:19.663 INFO:journalctl@ceph.osd.2.smithi178.stdout:Jul 14 17:45:19 smithi178 bash[158022]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-2 2023-07-14T17:45:20.974 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.2.service' 2023-07-14T17:45:21.015 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:45:21.016 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-14T17:45:21.016 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-14T17:45:21.017 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.3 2023-07-14T17:45:21.382 INFO:journalctl@ceph.osd.3.smithi088.stdout:Jul 14 17:45:21 smithi088 systemd[1]: Stopping Ceph osd.3 for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:45:21.690 INFO:journalctl@ceph.osd.3.smithi088.stdout:Jul 14 17:45:21 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-3[145776]: 2023-07-14T17:45:21.399+0000 7f723d746700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:45:21.691 INFO:journalctl@ceph.osd.3.smithi088.stdout:Jul 14 17:45:21 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-3[145776]: 2023-07-14T17:45:21.399+0000 7f723d746700 -1 osd.3 94 *** Got signal Terminated *** 2023-07-14T17:45:21.691 INFO:journalctl@ceph.osd.3.smithi088.stdout:Jul 14 17:45:21 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-3[145776]: 2023-07-14T17:45:21.399+0000 7f723d746700 -1 osd.3 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T17:45:25.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:25 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:25.568+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:26.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:26 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:26.591+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:27.156 INFO:journalctl@ceph.osd.3.smithi088.stdout:Jul 14 17:45:27 smithi088 bash[181081]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-3 2023-07-14T17:45:27.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:27 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:27.618+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:28.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:27 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:27.902+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:28.415 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:28 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:28.205+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:28.422 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.3.service' 2023-07-14T17:45:28.463 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:45:28.463 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-14T17:45:28.464 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-14T17:45:28.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.4 2023-07-14T17:45:28.825 INFO:journalctl@ceph.osd.4.smithi107.stdout:Jul 14 17:45:28 smithi107 systemd[1]: Stopping Ceph osd.4 for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:45:28.826 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:28 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:28.622+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:29.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:28 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:28.883+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:29.223 INFO:journalctl@ceph.osd.4.smithi107.stdout:Jul 14 17:45:28 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-4[139372]: 2023-07-14T17:45:28.843+0000 7fbdd13a9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:45:29.223 INFO:journalctl@ceph.osd.4.smithi107.stdout:Jul 14 17:45:28 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-4[139372]: 2023-07-14T17:45:28.843+0000 7fbdd13a9700 -1 osd.4 94 *** Got signal Terminated *** 2023-07-14T17:45:29.223 INFO:journalctl@ceph.osd.4.smithi107.stdout:Jul 14 17:45:28 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-4[139372]: 2023-07-14T17:45:28.843+0000 7fbdd13a9700 -1 osd.4 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T17:45:29.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:29 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:29.231+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:29.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:29 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:29.601+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:30.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:29 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:29.894+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:30.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:30 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:30.254+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:30.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:30 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:30.628+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:31.162 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:30 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:30.859+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:31.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:31 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:31.213+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:31.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:31 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:31.581+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:32.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:31 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:31.824+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:32.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:32 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:32.246+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:32.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:32 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:32.246+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:32.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:32 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:32.592+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:33.162 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:32 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:32.856+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:33.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:32 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:32.882+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:33.181 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:32 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:32.740+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.098706+0000 front 2023-07-14T17:45:10.098656+0000 (oldest deadline 2023-07-14T17:45:32.398701+0000) 2023-07-14T17:45:33.440 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:33 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:33.200+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:33.441 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:33 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:33.200+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:33.940 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:33 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:33.693+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.098706+0000 front 2023-07-14T17:45:10.098656+0000 (oldest deadline 2023-07-14T17:45:32.398701+0000) 2023-07-14T17:45:33.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:33 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:33.564+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:34.162 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:33 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:33.922+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:34.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:33 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:33.811+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:34.440 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:34 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:34.152+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:34.440 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:34 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:34.152+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:34.663 INFO:journalctl@ceph.osd.5.smithi178.stdout:Jul 14 17:45:34 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-5[140687]: 2023-07-14T17:45:34.322+0000 7f1d018d4700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.539012+0000 front 2023-07-14T17:45:10.538917+0000 (oldest deadline 2023-07-14T17:45:33.438923+0000) 2023-07-14T17:45:34.723 INFO:journalctl@ceph.osd.4.smithi107.stdout:Jul 14 17:45:34 smithi107 bash[156781]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-4 2023-07-14T17:45:34.724 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:34 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:34.550+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:35.162 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:34 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:34.904+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:35.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:34 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:34.841+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:35.170 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:34 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:34.720+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.098706+0000 front 2023-07-14T17:45:10.098656+0000 (oldest deadline 2023-07-14T17:45:32.398701+0000) 2023-07-14T17:45:35.440 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:35 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:35.190+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:35.441 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:35 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:35.190+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:35.663 INFO:journalctl@ceph.osd.5.smithi178.stdout:Jul 14 17:45:35 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-5[140687]: 2023-07-14T17:45:35.352+0000 7f1d018d4700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.539012+0000 front 2023-07-14T17:45:10.538917+0000 (oldest deadline 2023-07-14T17:45:33.438923+0000) 2023-07-14T17:45:35.723 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:35 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:35.546+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:35.723 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:35 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:35.546+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:35.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.4.service' 2023-07-14T17:45:35.843 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:45:35.844 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-14T17:45:35.844 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-14T17:45:35.844 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.5 2023-07-14T17:45:36.163 INFO:journalctl@ceph.osd.5.smithi178.stdout:Jul 14 17:45:35 smithi178 systemd[1]: Stopping Ceph osd.5 for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:45:36.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:35 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:35.916+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:36.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:35 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:35.791+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:36.190 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:35 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:35.718+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.098706+0000 front 2023-07-14T17:45:10.098656+0000 (oldest deadline 2023-07-14T17:45:32.398701+0000) 2023-07-14T17:45:36.662 INFO:journalctl@ceph.osd.5.smithi178.stdout:Jul 14 17:45:36 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-5[140687]: 2023-07-14T17:45:36.216+0000 7f1d072e1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:45:36.663 INFO:journalctl@ceph.osd.5.smithi178.stdout:Jul 14 17:45:36 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-5[140687]: 2023-07-14T17:45:36.216+0000 7f1d072e1700 -1 osd.5 94 *** Got signal Terminated *** 2023-07-14T17:45:36.663 INFO:journalctl@ceph.osd.5.smithi178.stdout:Jul 14 17:45:36 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-5[140687]: 2023-07-14T17:45:36.216+0000 7f1d072e1700 -1 osd.5 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T17:45:36.663 INFO:journalctl@ceph.osd.5.smithi178.stdout:Jul 14 17:45:36 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-5[140687]: 2023-07-14T17:45:36.349+0000 7f1d018d4700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.539012+0000 front 2023-07-14T17:45:10.538917+0000 (oldest deadline 2023-07-14T17:45:33.438923+0000) 2023-07-14T17:45:36.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:36 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:36.226+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:36.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:36 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:36.226+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:36.972 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:36 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:36.759+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:36.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:36 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:36.582+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:36.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:36 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:36.582+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:37.162 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:36 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:36.867+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:37.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:36 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:36.788+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:37.190 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:36 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:36.750+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.098706+0000 front 2023-07-14T17:45:10.098656+0000 (oldest deadline 2023-07-14T17:45:32.398701+0000) 2023-07-14T17:45:37.662 INFO:journalctl@ceph.osd.5.smithi178.stdout:Jul 14 17:45:37 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-5[140687]: 2023-07-14T17:45:37.394+0000 7f1d018d4700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.539012+0000 front 2023-07-14T17:45:10.538917+0000 (oldest deadline 2023-07-14T17:45:33.438923+0000) 2023-07-14T17:45:37.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:37 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:37.262+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:37.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:37 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:37.262+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:37.972 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:37 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:37.736+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:37.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:37 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:37.606+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:37.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:37 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:37.606+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:38.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:37 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:37.859+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:38.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:37 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:37.752+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:38.190 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:37 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:37.751+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.098706+0000 front 2023-07-14T17:45:10.098656+0000 (oldest deadline 2023-07-14T17:45:32.398701+0000) 2023-07-14T17:45:38.662 INFO:journalctl@ceph.osd.5.smithi178.stdout:Jul 14 17:45:38 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-5[140687]: 2023-07-14T17:45:38.380+0000 7f1d018d4700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.539012+0000 front 2023-07-14T17:45:10.538917+0000 (oldest deadline 2023-07-14T17:45:33.438923+0000) 2023-07-14T17:45:38.685 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:38 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:38.300+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:38.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:38 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:38.300+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:38.940 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:38 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:38.704+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.098706+0000 front 2023-07-14T17:45:10.098656+0000 (oldest deadline 2023-07-14T17:45:32.398701+0000) 2023-07-14T17:45:38.972 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:38 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:38.711+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:38.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:38 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:38.608+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:38.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:38 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:38.608+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:39.162 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:38 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:38.823+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:39.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:38 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:38.756+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:39.646 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:39 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:39.289+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:39.647 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:39 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:39.289+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:39.662 INFO:journalctl@ceph.osd.5.smithi178.stdout:Jul 14 17:45:39 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-5[140687]: 2023-07-14T17:45:39.338+0000 7f1d018d4700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.539012+0000 front 2023-07-14T17:45:10.538917+0000 (oldest deadline 2023-07-14T17:45:33.438923+0000) 2023-07-14T17:45:39.940 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:39 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:39.665+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.098706+0000 front 2023-07-14T17:45:10.098656+0000 (oldest deadline 2023-07-14T17:45:32.398701+0000) 2023-07-14T17:45:39.973 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:39 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:39.692+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:39.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:39 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:39.656+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:39.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:39 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:39.656+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:40.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:39 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:39.782+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:40.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:39 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:39.794+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:40.597 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:40 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:40.280+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:40.598 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:40 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:40.280+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:40.662 INFO:journalctl@ceph.osd.5.smithi178.stdout:Jul 14 17:45:40 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-5[140687]: 2023-07-14T17:45:40.299+0000 7f1d018d4700 -1 osd.5 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.539012+0000 front 2023-07-14T17:45:10.538917+0000 (oldest deadline 2023-07-14T17:45:33.438923+0000) 2023-07-14T17:45:40.940 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:40 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:40.617+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.098706+0000 front 2023-07-14T17:45:10.098656+0000 (oldest deadline 2023-07-14T17:45:32.398701+0000) 2023-07-14T17:45:40.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:40 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:40.687+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:40.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:40 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:40.688+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:40.973 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:40 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:40.678+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:40.973 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:40 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:40.678+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.327619+0000 front 2023-07-14T17:45:16.327403+0000 (oldest deadline 2023-07-14T17:45:39.827427+0000) 2023-07-14T17:45:41.162 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:40 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:40.750+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:41.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:40 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:40.767+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:41.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:40 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:40.767+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:41.562 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:41 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:41.317+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:41.563 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:41 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:41.317+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:41.940 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:41 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:41.581+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.098706+0000 front 2023-07-14T17:45:10.098656+0000 (oldest deadline 2023-07-14T17:45:32.398701+0000) 2023-07-14T17:45:41.940 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:41 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:41.581+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.600242+0000 front 2023-07-14T17:45:17.600192+0000 (oldest deadline 2023-07-14T17:45:41.100195+0000) 2023-07-14T17:45:41.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:41 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:41.691+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:41.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:41 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:41.691+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:41.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:41 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:41.691+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:41.974 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:41 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:41.648+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:41.974 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:41 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:41.648+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.327619+0000 front 2023-07-14T17:45:16.327403+0000 (oldest deadline 2023-07-14T17:45:39.827427+0000) 2023-07-14T17:45:41.985 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:41 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:41.768+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:41.986 INFO:journalctl@ceph.osd.5.smithi178.stdout:Jul 14 17:45:41 smithi178 bash[158295]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-5 2023-07-14T17:45:41.986 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:41 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:41.772+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:41.986 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:41 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:41.772+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:42.529 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:42 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:42.269+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:42.529 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:42 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:42.269+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:42.906 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:42 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:42.746+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:42.907 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:42 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:42.746+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:45:42.907 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:42 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:42.739+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:42.907 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:42 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:42.739+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:42.940 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:42 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:42.548+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.098706+0000 front 2023-07-14T17:45:10.098656+0000 (oldest deadline 2023-07-14T17:45:32.398701+0000) 2023-07-14T17:45:42.940 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:42 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:42.548+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.600242+0000 front 2023-07-14T17:45:17.600192+0000 (oldest deadline 2023-07-14T17:45:41.100195+0000) 2023-07-14T17:45:42.972 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:42 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:42.644+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:42.973 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:42 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:42.644+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.327619+0000 front 2023-07-14T17:45:16.327403+0000 (oldest deadline 2023-07-14T17:45:39.827427+0000) 2023-07-14T17:45:42.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:42 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:42.659+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:42.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:42 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:42.659+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:42.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:42 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:42.659+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:43.250 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.5.service' 2023-07-14T17:45:43.291 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:45:43.291 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-14T17:45:43.292 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-14T17:45:43.292 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.6 2023-07-14T17:45:43.548 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:43 smithi088 systemd[1]: Stopping Ceph osd.6 for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:45:43.549 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:43 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:43.243+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:43.550 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:43 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:43.243+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:43.941 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:43 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:43.567+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.098706+0000 front 2023-07-14T17:45:10.098656+0000 (oldest deadline 2023-07-14T17:45:32.398701+0000) 2023-07-14T17:45:43.941 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:43 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:43.567+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.600242+0000 front 2023-07-14T17:45:17.600192+0000 (oldest deadline 2023-07-14T17:45:41.100195+0000) 2023-07-14T17:45:43.941 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:43 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:43.670+0000 7f74f4ffb700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:45:43.942 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:43 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:43.670+0000 7f74f4ffb700 -1 osd.6 94 *** Got signal Terminated *** 2023-07-14T17:45:43.942 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:43 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:43.670+0000 7f74f4ffb700 -1 osd.6 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T17:45:43.972 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:43 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:43.625+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:43.973 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:43 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:43.625+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.327619+0000 front 2023-07-14T17:45:16.327403+0000 (oldest deadline 2023-07-14T17:45:39.827427+0000) 2023-07-14T17:45:43.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:43 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:43.629+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:43.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:43 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:43.629+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:43.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:43 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:43.629+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:44.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:43 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:43.724+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:44.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:43 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:43.724+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:45:44.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:43 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:43.744+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:44.165 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:43 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:43.744+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:44.531 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:44 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:44.236+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:44.531 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:44 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:44.236+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:44.531 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:44 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:44.236+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:45:44.940 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:44 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:44.550+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.098706+0000 front 2023-07-14T17:45:10.098656+0000 (oldest deadline 2023-07-14T17:45:32.398701+0000) 2023-07-14T17:45:44.940 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:44 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:44.550+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.600242+0000 front 2023-07-14T17:45:17.600192+0000 (oldest deadline 2023-07-14T17:45:41.100195+0000) 2023-07-14T17:45:44.972 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:44 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:44.587+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:44.973 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:44 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:44.587+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.327619+0000 front 2023-07-14T17:45:16.327403+0000 (oldest deadline 2023-07-14T17:45:39.827427+0000) 2023-07-14T17:45:44.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:44 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:44.629+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:44.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:44 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:44.629+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:44.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:44 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:44.629+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:45.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:44 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:44.748+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:45.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:44 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:44.748+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:45:45.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:44 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:44.739+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:45.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:44 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:44.739+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:45.482 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:45 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:45.210+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:45.482 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:45 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:45.210+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:45.482 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:45 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:45.210+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:45:45.940 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:45 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:45.501+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.098706+0000 front 2023-07-14T17:45:10.098656+0000 (oldest deadline 2023-07-14T17:45:32.398701+0000) 2023-07-14T17:45:45.940 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:45 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:45.501+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.600242+0000 front 2023-07-14T17:45:17.600192+0000 (oldest deadline 2023-07-14T17:45:41.100195+0000) 2023-07-14T17:45:45.973 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:45 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:45.603+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:45.973 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:45 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:45.603+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.327619+0000 front 2023-07-14T17:45:16.327403+0000 (oldest deadline 2023-07-14T17:45:39.827427+0000) 2023-07-14T17:45:45.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:45 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:45.622+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:45.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:45 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:45.622+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:45.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:45 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:45.622+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:46.162 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:45 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:45.721+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:46.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:45 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:45.721+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:45:46.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:45 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:45.730+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:46.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:45 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:45.730+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:46.486 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:46 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:46.243+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:46.486 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:46 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:46.243+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:46.486 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:46 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:46.243+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:45:46.912 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:46 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:46.674+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:46.913 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:46 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:46.674+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:45:46.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:46 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:46.771+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:46.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:46 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:46.771+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:46.940 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:46 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:46.505+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.098706+0000 front 2023-07-14T17:45:10.098656+0000 (oldest deadline 2023-07-14T17:45:32.398701+0000) 2023-07-14T17:45:46.940 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:46 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:46.505+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.600242+0000 front 2023-07-14T17:45:17.600192+0000 (oldest deadline 2023-07-14T17:45:41.100195+0000) 2023-07-14T17:45:46.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:46 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:46.632+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:46.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:46 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:46.632+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:46.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:46 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:46.632+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:46.974 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:46 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:46.571+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:46.974 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:46 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:46.571+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.327619+0000 front 2023-07-14T17:45:16.327403+0000 (oldest deadline 2023-07-14T17:45:39.827427+0000) 2023-07-14T17:45:47.440 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:47 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:47.203+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:47.441 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:47 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:47.203+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:47.441 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:47 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:47.203+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:45:47.913 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:47.642+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:47.913 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:47.642+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:45:47.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:47.776+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:47.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:47 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:47.776+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:47.940 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:47 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:47.529+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.098706+0000 front 2023-07-14T17:45:10.098656+0000 (oldest deadline 2023-07-14T17:45:32.398701+0000) 2023-07-14T17:45:47.941 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:47 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:47.529+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.600242+0000 front 2023-07-14T17:45:17.600192+0000 (oldest deadline 2023-07-14T17:45:41.100195+0000) 2023-07-14T17:45:47.941 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:47 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:47.529+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:21.100599+0000 front 2023-07-14T17:45:21.100546+0000 (oldest deadline 2023-07-14T17:45:47.000700+0000) 2023-07-14T17:45:47.972 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:47.532+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:47.973 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:47.532+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.327619+0000 front 2023-07-14T17:45:16.327403+0000 (oldest deadline 2023-07-14T17:45:39.827427+0000) 2023-07-14T17:45:47.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:47.620+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:47.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:47.620+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:47.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:47 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:47.620+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:48.510 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:48 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:48.235+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:48.510 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:48 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:48.235+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:48.510 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:48 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:48.235+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:45:48.912 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:48 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:48.667+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:48.913 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:48 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:48.667+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:45:48.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:48 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:48.801+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:48.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:48 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:48.801+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:48.940 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:48 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:48.529+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:10.098706+0000 front 2023-07-14T17:45:10.098656+0000 (oldest deadline 2023-07-14T17:45:32.398701+0000) 2023-07-14T17:45:48.940 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:48 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:48.529+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.600242+0000 front 2023-07-14T17:45:17.600192+0000 (oldest deadline 2023-07-14T17:45:41.100195+0000) 2023-07-14T17:45:48.941 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:48 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6[151184]: 2023-07-14T17:45:48.529+0000 7f74eeded700 -1 osd.6 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:21.100599+0000 front 2023-07-14T17:45:21.100546+0000 (oldest deadline 2023-07-14T17:45:47.000700+0000) 2023-07-14T17:45:48.972 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:48 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:48.579+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:48.973 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:48 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:48.579+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.327619+0000 front 2023-07-14T17:45:16.327403+0000 (oldest deadline 2023-07-14T17:45:39.827427+0000) 2023-07-14T17:45:48.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:48 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:48.669+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:48.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:48 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:48.669+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:48.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:48 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:48.669+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:49.206 INFO:journalctl@ceph.osd.6.smithi088.stdout:Jul 14 17:45:49 smithi088 bash[181350]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-6 2023-07-14T17:45:49.207 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:49 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:49.224+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:49.207 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:49 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:49.224+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:49.207 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:49 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:49.224+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:45:49.912 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:49 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:49.665+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:49.913 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:49 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:49.665+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:45:49.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:49 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:49.798+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:49.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:49 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:49.798+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:49.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:49 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:49.798+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:45:49.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:49 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:49.703+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:49.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:49 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:49.703+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:49.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:49 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:49.703+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:49.973 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:49 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:49.580+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:49.973 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:49 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:49.580+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.327619+0000 front 2023-07-14T17:45:16.327403+0000 (oldest deadline 2023-07-14T17:45:39.827427+0000) 2023-07-14T17:45:49.974 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:49 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:49.580+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:26.030196+0000 front 2023-07-14T17:45:26.030413+0000 (oldest deadline 2023-07-14T17:45:48.930230+0000) 2023-07-14T17:45:50.190 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:49 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:49.224+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:45:50.541 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:50 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:50.220+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:50.542 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:50 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:50.220+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:50.542 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:50 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:50.220+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:45:50.542 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:50 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:50.220+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:45:50.548 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.6.service' 2023-07-14T17:45:50.588 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:45:50.589 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-14T17:45:50.589 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-14T17:45:50.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.7 2023-07-14T17:45:50.945 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:50 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:50.562+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:50.945 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:50 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:50.562+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.327619+0000 front 2023-07-14T17:45:16.327403+0000 (oldest deadline 2023-07-14T17:45:39.827427+0000) 2023-07-14T17:45:50.945 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:50 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:50.562+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:26.030196+0000 front 2023-07-14T17:45:26.030413+0000 (oldest deadline 2023-07-14T17:45:48.930230+0000) 2023-07-14T17:45:50.946 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:50 smithi107 systemd[1]: Stopping Ceph osd.7 for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:45:50.946 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:50 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:50.734+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:50.946 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:50 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:50.734+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:50.946 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:50 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:50.734+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:50.947 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:50 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:50.734+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:45:51.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:50 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:50.698+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:51.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:50 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:50.699+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:45:51.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:50 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:50.833+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:51.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:50 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:50.833+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:51.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:50 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:50.833+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:45:51.223 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:50 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:50.963+0000 7f5af3108700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:45:51.223 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:50 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:50.963+0000 7f5af3108700 -1 osd.7 94 *** Got signal Terminated *** 2023-07-14T17:45:51.223 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:50 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:50.963+0000 7f5af3108700 -1 osd.7 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T17:45:51.440 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:51 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:51.198+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:51.440 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:51 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:51.198+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:51.441 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:51 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:51.198+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:45:51.441 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:51 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:51.198+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:45:51.972 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:51 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:51.581+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:51.973 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:51 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:51.581+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.327619+0000 front 2023-07-14T17:45:16.327403+0000 (oldest deadline 2023-07-14T17:45:39.827427+0000) 2023-07-14T17:45:51.973 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:51 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:51.581+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:26.030196+0000 front 2023-07-14T17:45:26.030413+0000 (oldest deadline 2023-07-14T17:45:48.930230+0000) 2023-07-14T17:45:51.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:51 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:51.702+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:51.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:51 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:51.702+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:51.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:51 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:51.702+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:51.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:51 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:51.702+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:45:52.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:51 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:51.690+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:52.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:51 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:51.690+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:45:52.164 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:51 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:51.690+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:21.956273+0000 front 2023-07-14T17:45:21.956120+0000 (oldest deadline 2023-07-14T17:45:51.356971+0000) 2023-07-14T17:45:52.165 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:51 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:51.838+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:52.165 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:51 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:51.838+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:52.165 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:51 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:51.838+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:45:52.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:52 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:52.248+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:52.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:52 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:52.248+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:52.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:52 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:52.248+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:45:52.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:52 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:52.248+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:45:52.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:52 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:52.661+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:52.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:52 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:52.661+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:52.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:52 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:52.661+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:52.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:52 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:52.661+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:45:52.974 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:52 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:52.568+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:52.974 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:52 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:52.568+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.327619+0000 front 2023-07-14T17:45:16.327403+0000 (oldest deadline 2023-07-14T17:45:39.827427+0000) 2023-07-14T17:45:52.974 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:52 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:52.568+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:26.030196+0000 front 2023-07-14T17:45:26.030413+0000 (oldest deadline 2023-07-14T17:45:48.930230+0000) 2023-07-14T17:45:53.162 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:52 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:52.704+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:53.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:52 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:52.704+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:45:53.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:52 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:52.704+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:21.956273+0000 front 2023-07-14T17:45:21.956120+0000 (oldest deadline 2023-07-14T17:45:51.356971+0000) 2023-07-14T17:45:53.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:52 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:52.789+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:53.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:52 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:52.789+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:53.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:52 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:52.789+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:45:53.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:53 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:53.252+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:53.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:53 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:53.252+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:53.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:53 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:53.252+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:45:53.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:53 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:53.252+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:45:53.912 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:53 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:53.660+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:53.913 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:53 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:53.660+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:45:53.913 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:53 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:53.660+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:21.956273+0000 front 2023-07-14T17:45:21.956120+0000 (oldest deadline 2023-07-14T17:45:51.356971+0000) 2023-07-14T17:45:53.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:53 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:53.791+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:53.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:53 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:53.791+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:53.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:53 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:53.791+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:45:53.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:53 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:53.667+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:53.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:53 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:53.667+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:53.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:53 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:53.667+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:53.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:53 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:53.667+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:45:53.973 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:53 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:53.591+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:53.974 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:53 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:53.591+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.327619+0000 front 2023-07-14T17:45:16.327403+0000 (oldest deadline 2023-07-14T17:45:39.827427+0000) 2023-07-14T17:45:53.974 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:53 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:53.591+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:26.030196+0000 front 2023-07-14T17:45:26.030413+0000 (oldest deadline 2023-07-14T17:45:48.930230+0000) 2023-07-14T17:45:54.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:54 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:54.281+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:54.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:54 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:54.281+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:54.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:54 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:54.281+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:45:54.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:54 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:54.281+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:45:54.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:54 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:54.649+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:54.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:54 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:54.649+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:54.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:54 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:54.649+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:54.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:54 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:54.649+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:45:54.974 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:54 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:54.567+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:54.974 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:54 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:54.567+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.327619+0000 front 2023-07-14T17:45:16.327403+0000 (oldest deadline 2023-07-14T17:45:39.827427+0000) 2023-07-14T17:45:54.974 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:54 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:54.567+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:26.030196+0000 front 2023-07-14T17:45:26.030413+0000 (oldest deadline 2023-07-14T17:45:48.930230+0000) 2023-07-14T17:45:55.162 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:54 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:54.696+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:55.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:54 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:54.696+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:45:55.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:54 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:54.696+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:21.956273+0000 front 2023-07-14T17:45:21.956120+0000 (oldest deadline 2023-07-14T17:45:51.356971+0000) 2023-07-14T17:45:55.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:54 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:54.787+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:55.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:54 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:54.787+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:55.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:54 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:54.787+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:45:55.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:55 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:55.319+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:55.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:55 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:55.319+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:55.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:55 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:55.319+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:45:55.692 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:55 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:55.319+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:45:55.692 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:55 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:55.319+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.450515+0000 front 2023-07-14T17:45:31.450670+0000 (oldest deadline 2023-07-14T17:45:54.950527+0000) 2023-07-14T17:45:55.973 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:55 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:55.594+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:11.026889+0000 front 2023-07-14T17:45:11.026783+0000 (oldest deadline 2023-07-14T17:45:36.326868+0000) 2023-07-14T17:45:55.973 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:55 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:55.594+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.327619+0000 front 2023-07-14T17:45:16.327403+0000 (oldest deadline 2023-07-14T17:45:39.827427+0000) 2023-07-14T17:45:55.973 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:55 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7[144877]: 2023-07-14T17:45:55.594+0000 7f5aecefa700 -1 osd.7 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:26.030196+0000 front 2023-07-14T17:45:26.030413+0000 (oldest deadline 2023-07-14T17:45:48.930230+0000) 2023-07-14T17:45:55.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:55 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:55.619+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:55.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:55 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:55.619+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:55.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:55 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:55.619+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:55.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:55 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:55.619+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:45:55.975 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:55 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:55.619+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:45:56.162 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:55 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:55.713+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:56.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:55 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:55.713+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:45:56.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:55 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:55.713+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:21.956273+0000 front 2023-07-14T17:45:21.956120+0000 (oldest deadline 2023-07-14T17:45:51.356971+0000) 2023-07-14T17:45:56.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:55 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:55.768+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:56.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:55 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:55.768+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:56.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:55 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:55.768+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:45:56.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:55 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:55.768+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:45:56.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:56 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:56.291+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:56.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:56 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:56.291+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:56.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:56 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:56.291+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:45:56.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:56 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:56.291+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:45:56.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:56 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:56.291+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.450515+0000 front 2023-07-14T17:45:31.450670+0000 (oldest deadline 2023-07-14T17:45:54.950527+0000) 2023-07-14T17:45:56.722 INFO:journalctl@ceph.osd.7.smithi107.stdout:Jul 14 17:45:56 smithi107 bash[157048]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-7 2023-07-14T17:45:56.723 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:56 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:56.622+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:56.723 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:56 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:56.622+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:56.723 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:56 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:56.622+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:56.724 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:56 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:56.622+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:45:56.724 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:56 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:56.622+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:45:57.162 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:56 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:56.730+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:57.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:56 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:56.730+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:45:57.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:56 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:56.730+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:21.956273+0000 front 2023-07-14T17:45:21.956120+0000 (oldest deadline 2023-07-14T17:45:51.356971+0000) 2023-07-14T17:45:57.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:56 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:56.808+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:57.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:56 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:56.808+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:57.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:56 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:56.808+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:45:57.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:56 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:56.808+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:45:57.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:57 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:57.335+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:57.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:57 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:57.335+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:57.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:57 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:57.335+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:45:57.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:57 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:57.335+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:45:57.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:57 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:57.335+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.450515+0000 front 2023-07-14T17:45:31.450670+0000 (oldest deadline 2023-07-14T17:45:54.950527+0000) 2023-07-14T17:45:57.869 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:57 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:57.588+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:57.870 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:57 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:57.588+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:57.870 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:57 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:57.588+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:57.870 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:57 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:57.588+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:45:57.871 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:57 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:57.588+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:45:57.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.7.service' 2023-07-14T17:45:57.916 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:45:57.916 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-14T17:45:57.917 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-07-14T17:45:57.917 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.8 2023-07-14T17:45:58.162 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:57 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:57.761+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:58.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:57 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:57.761+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:58.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:57 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:57.761+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:45:58.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:57 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:57.761+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:45:58.164 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:57 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:57.732+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:58.164 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:57 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:57.732+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:45:58.164 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:57 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:57.732+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:21.956273+0000 front 2023-07-14T17:45:21.956120+0000 (oldest deadline 2023-07-14T17:45:51.356971+0000) 2023-07-14T17:45:58.164 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:57 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:57.732+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.357556+0000 front 2023-07-14T17:45:31.357711+0000 (oldest deadline 2023-07-14T17:45:57.257516+0000) 2023-07-14T17:45:58.165 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:57 smithi178 systemd[1]: Stopping Ceph osd.8 for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:45:58.662 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:58 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:58.295+0000 7fa3a345e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:45:58.663 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:58 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:58.295+0000 7fa3a345e700 -1 osd.8 94 *** Got signal Terminated *** 2023-07-14T17:45:58.663 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:58 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:58.295+0000 7fa3a345e700 -1 osd.8 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T17:45:58.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:58 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:58.362+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:58.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:58 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:58.362+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:58.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:58 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:58.362+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:45:58.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:58 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:58.362+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:45:58.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:58 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:58.362+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.450515+0000 front 2023-07-14T17:45:31.450670+0000 (oldest deadline 2023-07-14T17:45:54.950527+0000) 2023-07-14T17:45:58.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:58 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:58.573+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:58.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:58 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:58.573+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:58.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:58 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:58.573+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:58.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:58 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:58.573+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:45:58.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:58 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:58.573+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:45:59.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:58 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:58.757+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:45:59.164 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:58 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:58.757+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:45:59.164 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:58 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:58.757+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:21.956273+0000 front 2023-07-14T17:45:21.956120+0000 (oldest deadline 2023-07-14T17:45:51.356971+0000) 2023-07-14T17:45:59.164 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:58 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:58.757+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.357556+0000 front 2023-07-14T17:45:31.357711+0000 (oldest deadline 2023-07-14T17:45:57.257516+0000) 2023-07-14T17:45:59.165 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:58 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:58.772+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:45:59.166 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:58 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:58.772+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:45:59.166 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:58 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:58.772+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:45:59.166 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:58 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:58.772+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:45:59.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:59 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:59.353+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:45:59.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:59 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:59.353+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:45:59.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:59 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:59.353+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:45:59.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:59 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:59.353+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:45:59.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:45:59 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:45:59.353+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.450515+0000 front 2023-07-14T17:45:31.450670+0000 (oldest deadline 2023-07-14T17:45:54.950527+0000) 2023-07-14T17:45:59.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:59 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:59.601+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:45:59.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:59 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:59.601+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:45:59.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:59 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:59.601+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:45:59.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:59 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:59.601+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:45:59.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:45:59 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:45:59.601+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:00.162 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:59 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:59.786+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:46:00.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:59 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:59.786+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:46:00.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:59 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:59.786+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:21.956273+0000 front 2023-07-14T17:45:21.956120+0000 (oldest deadline 2023-07-14T17:45:51.356971+0000) 2023-07-14T17:46:00.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:45:59 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:45:59.786+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.357556+0000 front 2023-07-14T17:45:31.357711+0000 (oldest deadline 2023-07-14T17:45:57.257516+0000) 2023-07-14T17:46:00.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:59 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:59.743+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:00.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:59 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:59.743+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:00.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:59 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:59.743+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:00.165 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:45:59 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:45:59.743+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:00.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:00 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:00.403+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:46:00.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:00 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:00.403+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:46:00.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:00 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:00.403+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:46:00.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:00 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:00.403+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:46:00.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:00 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:00.403+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.450515+0000 front 2023-07-14T17:45:31.450670+0000 (oldest deadline 2023-07-14T17:45:54.950527+0000) 2023-07-14T17:46:00.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:00 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:00.614+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:46:00.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:00 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:00.614+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:46:00.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:00 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:00.614+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:46:00.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:00 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:00.614+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:46:00.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:00 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:00.614+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:01.162 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:46:00 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:46:00.756+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:46:01.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:46:00 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:46:00.756+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:46:01.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:46:00 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:46:00.756+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:21.956273+0000 front 2023-07-14T17:45:21.956120+0000 (oldest deadline 2023-07-14T17:45:51.356971+0000) 2023-07-14T17:46:01.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:46:00 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:46:00.756+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.357556+0000 front 2023-07-14T17:45:31.357711+0000 (oldest deadline 2023-07-14T17:45:57.257516+0000) 2023-07-14T17:46:01.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:00 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:00.791+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:01.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:00 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:00.791+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:01.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:00 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:00.791+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:01.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:00 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:00.791+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:01.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:01 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:01.449+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:46:01.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:01 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:01.449+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:46:01.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:01 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:01.449+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:46:01.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:01 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:01.449+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:46:01.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:01 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:01.449+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.450515+0000 front 2023-07-14T17:45:31.450670+0000 (oldest deadline 2023-07-14T17:45:54.950527+0000) 2023-07-14T17:46:01.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:01 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:01.610+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:46:01.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:01 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:01.610+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:46:01.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:01 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:01.610+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:46:01.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:01 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:01.610+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:46:01.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:01 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:01.610+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:02.162 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:46:01 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:46:01.714+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:46:02.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:46:01 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:46:01.714+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:46:02.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:46:01 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:46:01.714+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:21.956273+0000 front 2023-07-14T17:45:21.956120+0000 (oldest deadline 2023-07-14T17:45:51.356971+0000) 2023-07-14T17:46:02.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:46:01 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:46:01.714+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.357556+0000 front 2023-07-14T17:45:31.357711+0000 (oldest deadline 2023-07-14T17:45:57.257516+0000) 2023-07-14T17:46:02.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:01 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:01.764+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:02.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:01 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:01.764+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:02.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:01 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:01.764+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:02.165 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:01 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:01.764+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:02.165 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:01 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:01.764+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:02.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:02 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:02.444+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:46:02.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:02 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:02.444+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:46:02.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:02 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:02.444+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:46:02.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:02 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:02.444+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:46:02.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:02 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:02.444+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.450515+0000 front 2023-07-14T17:45:31.450670+0000 (oldest deadline 2023-07-14T17:45:54.950527+0000) 2023-07-14T17:46:02.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:02 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:02.444+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.152381+0000 front 2023-07-14T17:45:40.152461+0000 (oldest deadline 2023-07-14T17:46:01.852330+0000) 2023-07-14T17:46:02.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:02 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:02.614+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:46:02.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:02 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:02.614+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:46:02.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:02 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:02.614+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:46:02.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:02 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:02.614+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:46:02.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:02 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:02.614+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:03.162 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:46:02 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:46:02.687+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.353947+0000 front 2023-07-14T17:45:07.353976+0000 (oldest deadline 2023-07-14T17:45:32.054063+0000) 2023-07-14T17:46:03.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:46:02 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:46:02.687+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:18.455691+0000 front 2023-07-14T17:45:18.455832+0000 (oldest deadline 2023-07-14T17:45:41.955801+0000) 2023-07-14T17:46:03.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:46:02 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:46:02.687+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:21.956273+0000 front 2023-07-14T17:45:21.956120+0000 (oldest deadline 2023-07-14T17:45:51.356971+0000) 2023-07-14T17:46:03.163 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:46:02 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:46:02.687+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.357556+0000 front 2023-07-14T17:45:31.357711+0000 (oldest deadline 2023-07-14T17:45:57.257516+0000) 2023-07-14T17:46:03.164 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:46:02 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8[146219]: 2023-07-14T17:46:02.687+0000 7fa39da51700 -1 osd.8 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:37.758549+0000 front 2023-07-14T17:45:37.758510+0000 (oldest deadline 2023-07-14T17:46:01.858603+0000) 2023-07-14T17:46:03.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:02 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:02.740+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:03.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:02 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:02.740+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:03.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:02 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:02.740+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:03.165 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:02 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:02.740+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:03.165 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:02 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:02.740+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:03.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:03 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:03.409+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:46:03.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:03 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:03.409+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:46:03.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:03 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:03.409+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:46:03.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:03 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:03.409+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:46:03.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:03 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:03.409+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.450515+0000 front 2023-07-14T17:45:31.450670+0000 (oldest deadline 2023-07-14T17:45:54.950527+0000) 2023-07-14T17:46:03.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:03 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:03.409+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.152381+0000 front 2023-07-14T17:45:40.152461+0000 (oldest deadline 2023-07-14T17:46:01.852330+0000) 2023-07-14T17:46:03.953 INFO:journalctl@ceph.osd.8.smithi178.stdout:Jul 14 17:46:03 smithi178 bash[158562]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-8 2023-07-14T17:46:03.953 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:03 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:03.744+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:03.954 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:03 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:03.744+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:03.954 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:03 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:03.744+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:03.954 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:03 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:03.744+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:03.954 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:03 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:03.744+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:03.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:03 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:03.585+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:46:03.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:03 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:03.585+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:46:03.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:03 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:03.585+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:46:03.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:03 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:03.585+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:46:03.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:03 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:03.585+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:03.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:03 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:03.585+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.209969+0000 front 2023-07-14T17:45:40.209905+0000 (oldest deadline 2023-07-14T17:46:03.109795+0000) 2023-07-14T17:46:04.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:04 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:04.387+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:46:04.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:04 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:04.387+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:46:04.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:04 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:04.387+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:46:04.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:04 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:04.387+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:46:04.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:04 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:04.387+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.450515+0000 front 2023-07-14T17:45:31.450670+0000 (oldest deadline 2023-07-14T17:45:54.950527+0000) 2023-07-14T17:46:04.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:04 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:04.387+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.152381+0000 front 2023-07-14T17:45:40.152461+0000 (oldest deadline 2023-07-14T17:46:01.852330+0000) 2023-07-14T17:46:04.908 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:04 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:04.712+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:04.909 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:04 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:04.712+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:04.909 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:04 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:04.712+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:04.909 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:04 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:04.712+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:04.910 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:04 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:04.712+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:04.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:04 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:04.552+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:46:04.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:04 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:04.552+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:46:04.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:04 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:04.552+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:46:04.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:04 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:04.552+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:46:04.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:04 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:04.552+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:04.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:04 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:04.552+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.209969+0000 front 2023-07-14T17:45:40.209905+0000 (oldest deadline 2023-07-14T17:46:03.109795+0000) 2023-07-14T17:46:05.212 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.8.service' 2023-07-14T17:46:05.251 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:46:05.252 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-07-14T17:46:05.252 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-07-14T17:46:05.253 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.9 2023-07-14T17:46:05.613 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:05 smithi088 systemd[1]: Stopping Ceph osd.9 for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:46:05.614 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:05 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:05.420+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:46:05.614 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:05 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:05.420+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:46:05.614 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:05 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:05.421+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:46:05.614 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:05 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:05.421+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:46:05.614 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:05 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:05.421+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.450515+0000 front 2023-07-14T17:45:31.450670+0000 (oldest deadline 2023-07-14T17:45:54.950527+0000) 2023-07-14T17:46:05.615 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:05 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:05.421+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.152381+0000 front 2023-07-14T17:45:40.152461+0000 (oldest deadline 2023-07-14T17:46:01.852330+0000) 2023-07-14T17:46:05.615 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:05 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:05.631+0000 7fb24d2ac700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:46:05.615 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:05 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:05.631+0000 7fb24d2ac700 -1 osd.9 94 *** Got signal Terminated *** 2023-07-14T17:46:05.615 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:05 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:05.631+0000 7fb24d2ac700 -1 osd.9 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T17:46:05.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:05 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:05.539+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:46:05.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:05 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:05.539+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:46:05.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:05 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:05.539+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:46:05.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:05 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:05.539+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:46:05.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:05 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:05.539+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:05.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:05 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:05.539+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.209969+0000 front 2023-07-14T17:45:40.209905+0000 (oldest deadline 2023-07-14T17:46:03.109795+0000) 2023-07-14T17:46:06.162 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:05 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:05.690+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:06.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:05 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:05.690+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:06.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:05 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:05.690+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:06.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:05 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:05.690+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:06.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:05 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:05.690+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:06.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:06 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:06.430+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:46:06.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:06 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:06.430+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:46:06.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:06 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:06.430+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:46:06.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:06 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:06.430+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:46:06.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:06 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:06.430+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.450515+0000 front 2023-07-14T17:45:31.450670+0000 (oldest deadline 2023-07-14T17:45:54.950527+0000) 2023-07-14T17:46:06.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:06 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:06.430+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.152381+0000 front 2023-07-14T17:45:40.152461+0000 (oldest deadline 2023-07-14T17:46:01.852330+0000) 2023-07-14T17:46:06.912 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:06 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:06.677+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:06.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:06 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:06.677+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:06.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:06 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:06.677+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:06.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:06 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:06.677+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:06.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:06 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:06.677+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:06.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:06 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:06.573+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:46:06.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:06 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:06.573+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:46:06.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:06 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:06.573+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:46:06.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:06 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:06.573+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:46:06.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:06 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:06.573+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:06.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:06 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:06.573+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.209969+0000 front 2023-07-14T17:45:40.209905+0000 (oldest deadline 2023-07-14T17:46:03.109795+0000) 2023-07-14T17:46:07.912 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:07 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:07.648+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:07.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:07 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:07.648+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:07.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:07 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:07.648+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:07.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:07 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:07.648+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:07.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:07 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:07.648+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:07.940 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:07 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:07.459+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:46:07.940 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:07 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:07.459+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:46:07.941 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:07 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:07.459+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:46:07.941 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:07 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:07.459+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:46:07.941 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:07 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:07.459+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.450515+0000 front 2023-07-14T17:45:31.450670+0000 (oldest deadline 2023-07-14T17:45:54.950527+0000) 2023-07-14T17:46:07.941 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:07 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:07.459+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.152381+0000 front 2023-07-14T17:45:40.152461+0000 (oldest deadline 2023-07-14T17:46:01.852330+0000) 2023-07-14T17:46:07.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:07 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:07.617+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:46:07.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:07 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:07.617+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:46:07.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:07 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:07.617+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:46:07.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:07 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:07.617+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:46:07.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:07 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:07.617+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:07.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:07 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:07.617+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.209969+0000 front 2023-07-14T17:45:40.209905+0000 (oldest deadline 2023-07-14T17:46:03.109795+0000) 2023-07-14T17:46:08.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:08 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:08.442+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:46:08.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:08 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:08.442+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:46:08.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:08 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:08.442+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:46:08.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:08 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:08.442+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:46:08.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:08 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:08.442+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.450515+0000 front 2023-07-14T17:45:31.450670+0000 (oldest deadline 2023-07-14T17:45:54.950527+0000) 2023-07-14T17:46:08.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:08 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:08.442+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.152381+0000 front 2023-07-14T17:45:40.152461+0000 (oldest deadline 2023-07-14T17:46:01.852330+0000) 2023-07-14T17:46:08.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:08 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:08.597+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:46:08.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:08 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:08.597+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:46:08.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:08 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:08.597+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:46:08.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:08 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:08.597+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:46:08.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:08 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:08.597+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:08.975 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:08 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:08.597+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.209969+0000 front 2023-07-14T17:45:40.209905+0000 (oldest deadline 2023-07-14T17:46:03.109795+0000) 2023-07-14T17:46:09.162 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:08 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:08.688+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:09.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:08 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:08.688+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:09.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:08 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:08.688+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:09.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:08 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:08.688+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:09.163 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:08 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:08.688+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:09.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:09 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:09.426+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:46:09.693 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:09 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:09.426+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:46:09.694 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:09 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:09.426+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:46:09.694 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:09 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:09.426+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:46:09.694 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:09 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:09.426+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.450515+0000 front 2023-07-14T17:45:31.450670+0000 (oldest deadline 2023-07-14T17:45:54.950527+0000) 2023-07-14T17:46:09.695 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:09 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:09.426+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.152381+0000 front 2023-07-14T17:45:40.152461+0000 (oldest deadline 2023-07-14T17:46:01.852330+0000) 2023-07-14T17:46:09.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:09 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:09.637+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:46:09.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:09 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:09.637+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:46:09.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:09 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:09.637+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:46:09.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:09 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:09.637+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:46:09.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:09 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:09.637+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:09.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:09 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:09.637+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.209969+0000 front 2023-07-14T17:45:40.209905+0000 (oldest deadline 2023-07-14T17:46:03.109795+0000) 2023-07-14T17:46:10.164 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:09 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:09.704+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:10.165 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:09 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:09.704+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:10.165 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:09 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:09.704+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:10.165 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:09 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:09.704+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:10.165 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:09 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:09.704+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:10.165 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:09 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:09.704+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:45.793185+0000 front 2023-07-14T17:45:45.793173+0000 (oldest deadline 2023-07-14T17:46:09.293068+0000) 2023-07-14T17:46:10.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:10 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:10.378+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.245547+0000 front 2023-07-14T17:45:02.245583+0000 (oldest deadline 2023-07-14T17:45:27.545748+0000) 2023-07-14T17:46:10.690 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:10 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:10.378+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:07.546239+0000 front 2023-07-14T17:45:07.546156+0000 (oldest deadline 2023-07-14T17:45:32.246294+0000) 2023-07-14T17:46:10.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:10 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:10.378+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:17.948376+0000 front 2023-07-14T17:45:17.948542+0000 (oldest deadline 2023-07-14T17:45:43.848464+0000) 2023-07-14T17:46:10.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:10 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:10.378+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:23.849070+0000 front 2023-07-14T17:45:23.848997+0000 (oldest deadline 2023-07-14T17:45:48.549286+0000) 2023-07-14T17:46:10.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:10 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:10.378+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:31.450515+0000 front 2023-07-14T17:45:31.450670+0000 (oldest deadline 2023-07-14T17:45:54.950527+0000) 2023-07-14T17:46:10.691 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:10 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9[156749]: 2023-07-14T17:46:10.378+0000 7fb24789f700 -1 osd.9 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.152381+0000 front 2023-07-14T17:45:40.152461+0000 (oldest deadline 2023-07-14T17:46:01.852330+0000) 2023-07-14T17:46:10.912 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:10 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:10.675+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:10.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:10 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:10.675+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:10.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:10 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:10.675+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:10.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:10 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:10.675+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:10.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:10 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:10.675+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:10.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:10 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:10.675+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:45.793185+0000 front 2023-07-14T17:45:45.793173+0000 (oldest deadline 2023-07-14T17:46:09.293068+0000) 2023-07-14T17:46:10.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:10 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:10.616+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:46:10.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:10 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:10.616+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:46:10.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:10 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:10.616+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:46:10.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:10 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:10.616+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:46:10.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:10 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:10.616+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:10.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:10 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:10.616+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.209969+0000 front 2023-07-14T17:45:40.209905+0000 (oldest deadline 2023-07-14T17:46:03.109795+0000) 2023-07-14T17:46:11.440 INFO:journalctl@ceph.osd.9.smithi088.stdout:Jul 14 17:46:11 smithi088 bash[181621]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-9 2023-07-14T17:46:11.912 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:11 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:11.631+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:11.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:11 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:11.631+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:11.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:11 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:11.631+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:11.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:11 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:11.631+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:11.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:11 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:11.631+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:11.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:11 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:11.631+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:45.793185+0000 front 2023-07-14T17:45:45.793173+0000 (oldest deadline 2023-07-14T17:46:09.293068+0000) 2023-07-14T17:46:11.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:11 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:11.596+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:46:11.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:11 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:11.596+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:46:11.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:11 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:11.596+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:46:11.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:11 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:11.596+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:46:11.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:11 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:11.596+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:11.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:11 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:11.596+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.209969+0000 front 2023-07-14T17:45:40.209905+0000 (oldest deadline 2023-07-14T17:46:03.109795+0000) 2023-07-14T17:46:12.600 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.9.service' 2023-07-14T17:46:12.639 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:46:12.640 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-07-14T17:46:12.640 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-07-14T17:46:12.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.10 2023-07-14T17:46:12.912 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:12 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:12.606+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:12.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:12 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:12.606+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:12.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:12 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:12.606+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:12.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:12 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:12.606+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:12.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:12 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:12.606+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:12.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:12 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:12.606+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:45.793185+0000 front 2023-07-14T17:45:45.793173+0000 (oldest deadline 2023-07-14T17:46:09.293068+0000) 2023-07-14T17:46:12.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:12 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:12.574+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:46:12.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:12 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:12.574+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:46:12.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:12 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:12.574+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:46:12.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:12 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:12.574+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:46:12.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:12 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:12.574+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:12.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:12 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:12.574+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.209969+0000 front 2023-07-14T17:45:40.209905+0000 (oldest deadline 2023-07-14T17:46:03.109795+0000) 2023-07-14T17:46:12.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:12 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:12.574+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:46.511556+0000 front 2023-07-14T17:45:46.511604+0000 (oldest deadline 2023-07-14T17:46:12.411593+0000) 2023-07-14T17:46:12.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:12 smithi107 systemd[1]: Stopping Ceph osd.10 for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:46:13.472 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:13 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:13.018+0000 7fa4a053f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:46:13.473 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:13 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:13.018+0000 7fa4a053f700 -1 osd.10 94 *** Got signal Terminated *** 2023-07-14T17:46:13.473 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:13 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:13.018+0000 7fa4a053f700 -1 osd.10 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T17:46:13.912 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:13 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:13.625+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:13.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:13 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:13.625+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:13.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:13 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:13.625+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:13.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:13 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:13.625+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:13.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:13 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:13.625+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:13.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:13 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:13.625+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:45.793185+0000 front 2023-07-14T17:45:45.793173+0000 (oldest deadline 2023-07-14T17:46:09.293068+0000) 2023-07-14T17:46:13.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:13 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:13.549+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:46:13.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:13 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:13.549+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:46:13.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:13 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:13.549+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:46:13.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:13 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:13.549+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:46:13.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:13 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:13.549+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:13.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:13 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:13.549+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.209969+0000 front 2023-07-14T17:45:40.209905+0000 (oldest deadline 2023-07-14T17:46:03.109795+0000) 2023-07-14T17:46:13.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:13 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:13.549+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:46.511556+0000 front 2023-07-14T17:45:46.511604+0000 (oldest deadline 2023-07-14T17:46:12.411593+0000) 2023-07-14T17:46:14.912 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:14 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:14.619+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:14.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:14 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:14.619+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:14.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:14 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:14.619+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:14.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:14 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:14.619+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:14.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:14 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:14.619+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:14.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:14 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:14.619+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:45.793185+0000 front 2023-07-14T17:45:45.793173+0000 (oldest deadline 2023-07-14T17:46:09.293068+0000) 2023-07-14T17:46:14.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:14 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:14.519+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:46:14.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:14 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:14.519+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:46:14.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:14 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:14.519+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:46:14.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:14 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:14.519+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:46:14.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:14 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:14.519+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:14.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:14 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:14.519+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.209969+0000 front 2023-07-14T17:45:40.209905+0000 (oldest deadline 2023-07-14T17:46:03.109795+0000) 2023-07-14T17:46:14.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:14 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:14.519+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:46.511556+0000 front 2023-07-14T17:45:46.511604+0000 (oldest deadline 2023-07-14T17:46:12.411593+0000) 2023-07-14T17:46:15.912 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:15 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:15.669+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:15.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:15 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:15.669+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:15.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:15 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:15.669+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:15.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:15 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:15.669+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:15.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:15 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:15.669+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:15.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:15 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:15.669+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:45.793185+0000 front 2023-07-14T17:45:45.793173+0000 (oldest deadline 2023-07-14T17:46:09.293068+0000) 2023-07-14T17:46:15.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:15 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:15.521+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:46:15.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:15 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:15.521+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:46:15.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:15 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:15.521+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:46:15.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:15 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:15.521+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:46:15.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:15 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:15.521+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:15.975 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:15 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:15.521+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.209969+0000 front 2023-07-14T17:45:40.209905+0000 (oldest deadline 2023-07-14T17:46:03.109795+0000) 2023-07-14T17:46:15.975 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:15 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:15.521+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:46.511556+0000 front 2023-07-14T17:45:46.511604+0000 (oldest deadline 2023-07-14T17:46:12.411593+0000) 2023-07-14T17:46:16.912 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:16 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:16.634+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:16.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:16 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:16.634+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:16.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:16 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:16.634+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:16.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:16 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:16.634+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:16.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:16 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:16.634+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:16.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:16 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:16.634+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:45.793185+0000 front 2023-07-14T17:45:45.793173+0000 (oldest deadline 2023-07-14T17:46:09.293068+0000) 2023-07-14T17:46:16.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:16 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:16.552+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:46:16.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:16 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:16.552+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:46:16.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:16 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:16.552+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:46:16.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:16 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:16.552+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:46:16.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:16 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:16.552+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:16.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:16 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:16.552+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.209969+0000 front 2023-07-14T17:45:40.209905+0000 (oldest deadline 2023-07-14T17:46:03.109795+0000) 2023-07-14T17:46:16.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:16 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:16.552+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:46.511556+0000 front 2023-07-14T17:45:46.511604+0000 (oldest deadline 2023-07-14T17:46:12.411593+0000) 2023-07-14T17:46:17.912 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:17 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:17.651+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:17.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:17 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:17.651+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:17.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:17 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:17.651+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:17.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:17 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:17.651+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:17.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:17 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:17.651+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:17.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:17 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:17.651+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:45.793185+0000 front 2023-07-14T17:45:45.793173+0000 (oldest deadline 2023-07-14T17:46:09.293068+0000) 2023-07-14T17:46:17.972 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:17 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:17.545+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:44:59.403899+0000 front 2023-07-14T17:44:59.403745+0000 (oldest deadline 2023-07-14T17:45:24.703900+0000) 2023-07-14T17:46:17.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:17 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:17.545+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6804 osd.1 since back 2023-07-14T17:45:09.805666+0000 front 2023-07-14T17:45:09.805717+0000 (oldest deadline 2023-07-14T17:45:35.105852+0000) 2023-07-14T17:46:17.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:17 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:17.545+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:16.806929+0000 front 2023-07-14T17:45:16.807116+0000 (oldest deadline 2023-07-14T17:45:41.507060+0000) 2023-07-14T17:46:17.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:17 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:17.545+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.607958+0000 front 2023-07-14T17:45:25.607979+0000 (oldest deadline 2023-07-14T17:45:50.308032+0000) 2023-07-14T17:46:17.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:17 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:17.545+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:30.308352+0000 front 2023-07-14T17:45:30.308402+0000 (oldest deadline 2023-07-14T17:45:55.008412+0000) 2023-07-14T17:46:17.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:17 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:17.545+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:40.209969+0000 front 2023-07-14T17:45:40.209905+0000 (oldest deadline 2023-07-14T17:46:03.109795+0000) 2023-07-14T17:46:17.974 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:17 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10[150315]: 2023-07-14T17:46:17.545+0000 7fa499b30700 -1 osd.10 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:46.511556+0000 front 2023-07-14T17:45:46.511604+0000 (oldest deadline 2023-07-14T17:46:12.411593+0000) 2023-07-14T17:46:18.912 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:18 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:18.652+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:18.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:18 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:18.652+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:18.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:18 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:18.652+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:18.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:18 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:18.652+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:18.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:18 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:18.652+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:18.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:18 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:18.652+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:45.793185+0000 front 2023-07-14T17:45:45.793173+0000 (oldest deadline 2023-07-14T17:46:09.293068+0000) 2023-07-14T17:46:18.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:18 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:18.652+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6820 osd.7 since back 2023-07-14T17:45:53.294562+0000 front 2023-07-14T17:45:53.294501+0000 (oldest deadline 2023-07-14T17:46:17.894928+0000) 2023-07-14T17:46:18.973 INFO:journalctl@ceph.osd.10.smithi107.stdout:Jul 14 17:46:18 smithi107 bash[157331]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-10 2023-07-14T17:46:19.912 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:19 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:19.634+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:19.912 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:19 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:19.634+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:19.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:19 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:19.634+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:19.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:19 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:19.634+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:19.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:19 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:19.634+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:19.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:19 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:19.634+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:45.793185+0000 front 2023-07-14T17:45:45.793173+0000 (oldest deadline 2023-07-14T17:46:09.293068+0000) 2023-07-14T17:46:19.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:19 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:19.634+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6820 osd.7 since back 2023-07-14T17:45:53.294562+0000 front 2023-07-14T17:45:53.294501+0000 (oldest deadline 2023-07-14T17:46:17.894928+0000) 2023-07-14T17:46:20.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.10.service' 2023-07-14T17:46:20.114 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:46:20.115 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-07-14T17:46:20.115 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-07-14T17:46:20.116 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.11 2023-07-14T17:46:20.412 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:20 smithi178 systemd[1]: Stopping Ceph osd.11 for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:46:20.912 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:20 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:20.482+0000 7ffa64157700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:46:20.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:20 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:20.482+0000 7ffa64157700 -1 osd.11 94 *** Got signal Terminated *** 2023-07-14T17:46:20.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:20 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:20.482+0000 7ffa64157700 -1 osd.11 94 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T17:46:20.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:20 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:20.616+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:20.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:20 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:20.616+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:20.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:20 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:20.616+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:20.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:20 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:20.616+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:20.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:20 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:20.616+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:20.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:20 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:20.616+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:45.793185+0000 front 2023-07-14T17:45:45.793173+0000 (oldest deadline 2023-07-14T17:46:09.293068+0000) 2023-07-14T17:46:20.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:20 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:20.616+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6820 osd.7 since back 2023-07-14T17:45:53.294562+0000 front 2023-07-14T17:45:53.294501+0000 (oldest deadline 2023-07-14T17:46:17.894928+0000) 2023-07-14T17:46:21.912 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:21 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:21.602+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:21.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:21 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:21.602+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:21.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:21 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:21.602+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:21.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:21 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:21.602+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:21.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:21 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:21.602+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:21.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:21 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:21.602+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:45.793185+0000 front 2023-07-14T17:45:45.793173+0000 (oldest deadline 2023-07-14T17:46:09.293068+0000) 2023-07-14T17:46:21.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:21 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:21.602+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6820 osd.7 since back 2023-07-14T17:45:53.294562+0000 front 2023-07-14T17:45:53.294501+0000 (oldest deadline 2023-07-14T17:46:17.894928+0000) 2023-07-14T17:46:22.912 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:22 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:22.639+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:22.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:22 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:22.639+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:22.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:22 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:22.639+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:22.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:22 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:22.639+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:22.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:22 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:22.639+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:22.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:22 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:22.639+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:45.793185+0000 front 2023-07-14T17:45:45.793173+0000 (oldest deadline 2023-07-14T17:46:09.293068+0000) 2023-07-14T17:46:22.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:22 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:22.639+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6820 osd.7 since back 2023-07-14T17:45:53.294562+0000 front 2023-07-14T17:45:53.294501+0000 (oldest deadline 2023-07-14T17:46:17.894928+0000) 2023-07-14T17:46:23.912 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:23 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:23.593+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:23.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:23 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:23.593+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:23.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:23 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:23.593+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:23.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:23 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:23.593+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:23.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:23 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:23.593+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:23.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:23 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:23.593+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:45.793185+0000 front 2023-07-14T17:45:45.793173+0000 (oldest deadline 2023-07-14T17:46:09.293068+0000) 2023-07-14T17:46:23.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:23 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:23.593+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6820 osd.7 since back 2023-07-14T17:45:53.294562+0000 front 2023-07-14T17:45:53.294501+0000 (oldest deadline 2023-07-14T17:46:17.894928+0000) 2023-07-14T17:46:24.912 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:24 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:24.596+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6806 osd.0 since back 2023-07-14T17:45:02.785713+0000 front 2023-07-14T17:45:02.785760+0000 (oldest deadline 2023-07-14T17:45:27.485797+0000) 2023-07-14T17:46:24.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:24 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:24.596+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6804 osd.2 since back 2023-07-14T17:45:15.889483+0000 front 2023-07-14T17:45:15.889452+0000 (oldest deadline 2023-07-14T17:45:40.589645+0000) 2023-07-14T17:46:24.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:24 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:24.596+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2023-07-14T17:45:25.290443+0000 front 2023-07-14T17:45:25.290483+0000 (oldest deadline 2023-07-14T17:45:49.390596+0000) 2023-07-14T17:46:24.913 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:24 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:24.596+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6812 osd.4 since back 2023-07-14T17:45:29.391158+0000 front 2023-07-14T17:45:29.391070+0000 (oldest deadline 2023-07-14T17:45:55.291274+0000) 2023-07-14T17:46:24.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:24 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:24.596+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.178:6812 osd.5 since back 2023-07-14T17:45:39.992373+0000 front 2023-07-14T17:45:39.992300+0000 (oldest deadline 2023-07-14T17:46:01.692321+0000) 2023-07-14T17:46:24.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:24 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:24.596+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.88:6822 osd.6 since back 2023-07-14T17:45:45.793185+0000 front 2023-07-14T17:45:45.793173+0000 (oldest deadline 2023-07-14T17:46:09.293068+0000) 2023-07-14T17:46:24.914 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:24 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11[151719]: 2023-07-14T17:46:24.596+0000 7ffa5df49700 -1 osd.11 94 heartbeat_check: no reply from 172.21.15.107:6820 osd.7 since back 2023-07-14T17:45:53.294562+0000 front 2023-07-14T17:45:53.294501+0000 (oldest deadline 2023-07-14T17:46:17.894928+0000) 2023-07-14T17:46:26.413 INFO:journalctl@ceph.osd.11.smithi178.stdout:Jul 14 17:46:26 smithi178 bash[158826]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-osd-11 2023-07-14T17:46:27.517 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@osd.11.service' 2023-07-14T17:46:27.557 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:46:27.557 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-07-14T17:46:27.558 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-07-14T17:46:27.558 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.a 2023-07-14T17:46:27.907 INFO:journalctl@ceph.mds.a.smithi088.stdout:Jul 14 17:46:27 smithi088 systemd[1]: Stopping Ceph mds.a for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:46:28.189 INFO:journalctl@ceph.mds.a.smithi088.stdout:Jul 14 17:46:27 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-a[162115]: 2023-07-14T17:46:27.926+0000 7f5b79440700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:46:28.190 INFO:journalctl@ceph.mds.a.smithi088.stdout:Jul 14 17:46:27 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-a[162115]: 2023-07-14T17:46:27.926+0000 7f5b79440700 -1 mds.a *** got signal Terminated *** 2023-07-14T17:46:32.386 INFO:journalctl@ceph.mds.a.smithi088.stdout:Jul 14 17:46:32 smithi088 bash[181884]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-a 2023-07-14T17:46:32.392 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.a.service' 2023-07-14T17:46:32.426 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:46:32.427 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-07-14T17:46:32.427 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-07-14T17:46:32.428 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.d 2023-07-14T17:46:32.690 INFO:journalctl@ceph.mds.d.smithi088.stdout:Jul 14 17:46:32 smithi088 systemd[1]: Stopping Ceph mds.d for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:46:33.190 INFO:journalctl@ceph.mds.d.smithi088.stdout:Jul 14 17:46:32 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-d[161117]: 2023-07-14T17:46:32.833+0000 7fa863c46700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:46:33.190 INFO:journalctl@ceph.mds.d.smithi088.stdout:Jul 14 17:46:32 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-d[161117]: 2023-07-14T17:46:32.833+0000 7fa863c46700 -1 mds.d *** got signal Terminated *** 2023-07-14T17:46:35.939 INFO:journalctl@ceph.mds.d.smithi088.stdout:Jul 14 17:46:35 smithi088 bash[182027]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-d 2023-07-14T17:46:35.953 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.d.service' 2023-07-14T17:46:35.988 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:46:35.989 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-07-14T17:46:35.989 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-07-14T17:46:35.990 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.g 2023-07-14T17:46:36.373 INFO:journalctl@ceph.mds.g.smithi088.stdout:Jul 14 17:46:36 smithi088 systemd[1]: Stopping Ceph mds.g for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:46:36.690 INFO:journalctl@ceph.mds.g.smithi088.stdout:Jul 14 17:46:36 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-g[161729]: 2023-07-14T17:46:36.390+0000 7fe793ce8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:46:36.690 INFO:journalctl@ceph.mds.g.smithi088.stdout:Jul 14 17:46:36 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-g[161729]: 2023-07-14T17:46:36.390+0000 7fe793ce8700 -1 mds.g *** got signal Terminated *** 2023-07-14T17:46:40.138 INFO:journalctl@ceph.mds.g.smithi088.stdout:Jul 14 17:46:40 smithi088 bash[182169]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-g 2023-07-14T17:46:40.324 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.g.service' 2023-07-14T17:46:40.357 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:46:40.358 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-07-14T17:46:40.358 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-07-14T17:46:40.359 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.j 2023-07-14T17:46:40.690 INFO:journalctl@ceph.mds.j.smithi088.stdout:Jul 14 17:46:40 smithi088 systemd[1]: Stopping Ceph mds.j for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:46:41.190 INFO:journalctl@ceph.mds.j.smithi088.stdout:Jul 14 17:46:40 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-j[160755]: 2023-07-14T17:46:40.791+0000 7f915460e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:46:41.190 INFO:journalctl@ceph.mds.j.smithi088.stdout:Jul 14 17:46:40 smithi088 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-j[160755]: 2023-07-14T17:46:40.791+0000 7f915460e700 -1 mds.j *** got signal Terminated *** 2023-07-14T17:46:45.190 INFO:journalctl@ceph.mds.j.smithi088.stdout:Jul 14 17:46:44 smithi088 bash[182309]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-j 2023-07-14T17:46:45.280 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.j.service' 2023-07-14T17:46:45.318 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:46:45.319 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-07-14T17:46:45.319 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-07-14T17:46:45.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.b 2023-07-14T17:46:45.675 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:46:45 smithi107 systemd[1]: Stopping Ceph mds.b for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:46:45.972 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:46:45 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-b[154079]: 2023-07-14T17:46:45.693+0000 7f0cdabae700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:46:45.973 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:46:45 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-b[154079]: 2023-07-14T17:46:45.693+0000 7f0cdabae700 -1 mds.b *** got signal Terminated *** 2023-07-14T17:46:48.137 INFO:journalctl@ceph.mds.b.smithi107.stdout:Jul 14 17:46:48 smithi107 bash[157596]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-b 2023-07-14T17:46:48.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.b.service' 2023-07-14T17:46:48.364 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:46:48.366 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-07-14T17:46:48.366 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-07-14T17:46:48.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.e 2023-07-14T17:46:48.722 INFO:journalctl@ceph.mds.e.smithi107.stdout:Jul 14 17:46:48 smithi107 systemd[1]: Stopping Ceph mds.e for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:46:49.222 INFO:journalctl@ceph.mds.e.smithi107.stdout:Jul 14 17:46:48 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-e[153402]: 2023-07-14T17:46:48.792+0000 7fca98c83700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:46:49.222 INFO:journalctl@ceph.mds.e.smithi107.stdout:Jul 14 17:46:48 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-e[153402]: 2023-07-14T17:46:48.792+0000 7fca98c83700 -1 mds.e *** got signal Terminated *** 2023-07-14T17:46:52.112 INFO:journalctl@ceph.mds.e.smithi107.stdout:Jul 14 17:46:52 smithi107 bash[157734]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-e 2023-07-14T17:46:52.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.e.service' 2023-07-14T17:46:52.332 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:46:52.332 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-07-14T17:46:52.333 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-07-14T17:46:52.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.h 2023-07-14T17:46:52.714 INFO:journalctl@ceph.mds.h.smithi107.stdout:Jul 14 17:46:52 smithi107 systemd[1]: Stopping Ceph mds.h for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:46:52.972 INFO:journalctl@ceph.mds.h.smithi107.stdout:Jul 14 17:46:52 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-h[153740]: 2023-07-14T17:46:52.733+0000 7f0e585d6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:46:52.972 INFO:journalctl@ceph.mds.h.smithi107.stdout:Jul 14 17:46:52 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-h[153740]: 2023-07-14T17:46:52.733+0000 7f0e585d6700 -1 mds.h *** got signal Terminated *** 2023-07-14T17:46:56.222 INFO:journalctl@ceph.mds.h.smithi107.stdout:Jul 14 17:46:55 smithi107 bash[157868]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-h 2023-07-14T17:46:56.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.h.service' 2023-07-14T17:46:56.347 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:46:56.347 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-07-14T17:46:56.348 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-07-14T17:46:56.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.k 2023-07-14T17:46:56.722 INFO:journalctl@ceph.mds.k.smithi107.stdout:Jul 14 17:46:56 smithi107 systemd[1]: Stopping Ceph mds.k for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:46:57.222 INFO:journalctl@ceph.mds.k.smithi107.stdout:Jul 14 17:46:56 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-k[153063]: 2023-07-14T17:46:56.747+0000 7f6d33c9f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:46:57.222 INFO:journalctl@ceph.mds.k.smithi107.stdout:Jul 14 17:46:56 smithi107 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-k[153063]: 2023-07-14T17:46:56.747+0000 7f6d33c9f700 -1 mds.k *** got signal Terminated *** 2023-07-14T17:47:00.299 INFO:journalctl@ceph.mds.k.smithi107.stdout:Jul 14 17:47:00 smithi107 bash[158012]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-k 2023-07-14T17:47:00.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.k.service' 2023-07-14T17:47:00.425 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:47:00.426 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-07-14T17:47:00.426 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-07-14T17:47:00.427 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.c 2023-07-14T17:47:00.773 INFO:journalctl@ceph.mds.c.smithi178.stdout:Jul 14 17:47:00 smithi178 systemd[1]: Stopping Ceph mds.c for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:47:01.162 INFO:journalctl@ceph.mds.c.smithi178.stdout:Jul 14 17:47:00 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-c[155243]: 2023-07-14T17:47:00.792+0000 7f84e7342700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:47:01.162 INFO:journalctl@ceph.mds.c.smithi178.stdout:Jul 14 17:47:00 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-c[155243]: 2023-07-14T17:47:00.792+0000 7f84e7342700 -1 mds.c *** got signal Terminated *** 2023-07-14T17:47:04.337 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.c.service' 2023-07-14T17:47:04.373 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:47:04.374 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-07-14T17:47:04.374 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-07-14T17:47:04.374 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.f 2023-07-14T17:47:04.662 INFO:journalctl@ceph.mds.f.smithi178.stdout:Jul 14 17:47:04 smithi178 systemd[1]: Stopping Ceph mds.f for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:47:05.162 INFO:journalctl@ceph.mds.f.smithi178.stdout:Jul 14 17:47:04 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-f[154891]: 2023-07-14T17:47:04.800+0000 7fdfdf49c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:47:05.162 INFO:journalctl@ceph.mds.f.smithi178.stdout:Jul 14 17:47:04 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-f[154891]: 2023-07-14T17:47:04.800+0000 7fdfdf49c700 -1 mds.f *** got signal Terminated *** 2023-07-14T17:47:08.404 INFO:journalctl@ceph.mds.f.smithi178.stdout:Jul 14 17:47:08 smithi178 bash[159238]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-f 2023-07-14T17:47:08.410 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.f.service' 2023-07-14T17:47:08.446 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:47:08.446 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-07-14T17:47:08.447 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-07-14T17:47:08.447 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.i 2023-07-14T17:47:08.662 INFO:journalctl@ceph.mds.i.smithi178.stdout:Jul 14 17:47:08 smithi178 systemd[1]: Stopping Ceph mds.i for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:47:09.162 INFO:journalctl@ceph.mds.i.smithi178.stdout:Jul 14 17:47:08 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-i[155583]: 2023-07-14T17:47:08.832+0000 7f4831cdd700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:47:09.162 INFO:journalctl@ceph.mds.i.smithi178.stdout:Jul 14 17:47:08 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-i[155583]: 2023-07-14T17:47:08.832+0000 7f4831cdd700 -1 mds.i *** got signal Terminated *** 2023-07-14T17:47:12.376 INFO:journalctl@ceph.mds.i.smithi178.stdout:Jul 14 17:47:12 smithi178 bash[159375]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-i 2023-07-14T17:47:12.380 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.i.service' 2023-07-14T17:47:12.413 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:47:12.414 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-07-14T17:47:12.414 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-07-14T17:47:12.415 DEBUG:teuthology.orchestra.run.smithi178:> sudo systemctl stop ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.l 2023-07-14T17:47:12.662 INFO:journalctl@ceph.mds.l.smithi178.stdout:Jul 14 17:47:12 smithi178 systemd[1]: Stopping Ceph mds.l for b462a556-226c-11ee-9b33-001a4aab830c... 2023-07-14T17:47:13.162 INFO:journalctl@ceph.mds.l.smithi178.stdout:Jul 14 17:47:12 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-l[154546]: 2023-07-14T17:47:12.813+0000 7fd3c5ab9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T17:47:13.163 INFO:journalctl@ceph.mds.l.smithi178.stdout:Jul 14 17:47:12 smithi178 ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-l[154546]: 2023-07-14T17:47:12.813+0000 7fd3c5ab9700 -1 mds.l *** got signal Terminated *** 2023-07-14T17:47:16.250 INFO:journalctl@ceph.mds.l.smithi178.stdout:Jul 14 17:47:16 smithi178 bash[159522]: ceph-b462a556-226c-11ee-9b33-001a4aab830c-mds-l 2023-07-14T17:47:16.334 DEBUG:teuthology.orchestra.run.smithi178:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b462a556-226c-11ee-9b33-001a4aab830c@mds.l.service' 2023-07-14T17:47:16.368 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T17:47:16.368 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-07-14T17:47:16.369 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b462a556-226c-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T17:47:16.667 INFO:teuthology.orchestra.run.smithi088.stdout:Deleting cluster with fsid: b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:47:18.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b462a556-226c-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T17:47:18.541 INFO:teuthology.orchestra.run.smithi107.stdout:Deleting cluster with fsid: b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:47:20.087 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b462a556-226c-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T17:47:20.388 INFO:teuthology.orchestra.run.smithi178.stdout:Deleting cluster with fsid: b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:47:21.952 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T17:47:21.989 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T17:47:22.026 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T17:47:22.056 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-14T17:47:22.063 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337974/remote/smithi088/crash 2023-07-14T17:47:22.064 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T17:47:22.125 INFO:teuthology.orchestra.run.smithi088.stderr:tar: /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T17:47:22.125 INFO:teuthology.orchestra.run.smithi088.stderr:tar: Error is not recoverable: exiting now 2023-07-14T17:47:22.127 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337974/remote/smithi107/crash 2023-07-14T17:47:22.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T17:47:22.199 INFO:teuthology.orchestra.run.smithi107.stderr:tar: /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T17:47:22.200 INFO:teuthology.orchestra.run.smithi107.stderr:tar: Error is not recoverable: exiting now 2023-07-14T17:47:22.203 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337974/remote/smithi178/crash 2023-07-14T17:47:22.210 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T17:47:22.274 INFO:teuthology.orchestra.run.smithi178.stderr:tar: /var/lib/ceph/b462a556-226c-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T17:47:22.274 INFO:teuthology.orchestra.run.smithi178.stderr:tar: Error is not recoverable: exiting now 2023-07-14T17:47:22.275 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-14T17:47:22.276 DEBUG:teuthology.orchestra.run.smithi088:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-07-14T17:47:22.310 INFO:teuthology.orchestra.run.smithi088.stderr:grep: /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph.log: No such file or directory 2023-07-14T17:47:22.312 INFO:tasks.cephadm:Compressing logs... 2023-07-14T17:47:22.313 DEBUG:teuthology.orchestra.run.smithi088:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T17:47:22.355 DEBUG:teuthology.orchestra.run.smithi107:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T17:47:22.357 DEBUG:teuthology.orchestra.run.smithi178:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T17:47:22.381 INFO:teuthology.orchestra.run.smithi088.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log : No such file or directory 2023-07-14T17:47:22.382 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-07-14T17:47:22.382 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.0.log 2023-07-14T17:47:22.383 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.g.log 2023-07-14T17:47:22.385 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.a.log 2023-07-14T17:47:22.385 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.6.log 2023-07-14T17:47:22.385 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.3.log 2023-07-14T17:47:22.386 INFO:teuthology.orchestra.run.smithi178.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-07-14T17:47:22.386 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T17:47:22.387 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.5.log 2023-07-14T17:47:22.387 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.2.log 2023-07-14T17:47:22.388 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.11.log 2023-07-14T17:47:22.388 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.5.log: /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mgr.z.log 2023-07-14T17:47:22.388 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.8.log 2023-07-14T17:47:22.388 INFO:teuthology.orchestra.run.smithi178.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T17:47:22.389 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.f.log 2023-07-14T17:47:22.389 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mgr.z.log: /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.c.log 2023-07-14T17:47:22.390 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.f.log: /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.c.log: 91.7% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mgr.z.log.gz 2023-07-14T17:47:22.391 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.6.log: /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.d.log 2023-07-14T17:47:22.391 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mgr.x.log 2023-07-14T17:47:22.391 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mgr.x.log: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T17:47:22.392 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.9.log 2023-07-14T17:47:22.392 INFO:teuthology.orchestra.run.smithi088.stderr: 94.8% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.d.log.gz 2023-07-14T17:47:22.393 INFO:teuthology.orchestra.run.smithi107.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T17:47:22.393 INFO:teuthology.orchestra.run.smithi107.stderr:: No such file or directory 2023-07-14T17:47:22.393 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.10.log 2023-07-14T17:47:22.394 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.h.log 2023-07-14T17:47:22.394 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.e.log 2023-07-14T17:47:22.394 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.10.log: /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.4.log 2023-07-14T17:47:22.394 INFO:teuthology.orchestra.run.smithi107.stderr: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T17:47:22.395 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.i.log 2023-07-14T17:47:22.396 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mon.c.log 2023-07-14T17:47:22.396 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T17:47:22.396 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.j.log 2023-07-14T17:47:22.397 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.1.log 2023-07-14T17:47:22.397 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.k.log 2023-07-14T17:47:22.398 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T17:47:22.398 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.k.log: /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.7.log 2023-07-14T17:47:22.399 INFO:teuthology.orchestra.run.smithi107.stderr: 91.5% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T17:47:22.399 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mon.b.log 2023-07-14T17:47:22.399 INFO:teuthology.orchestra.run.smithi088.stderr: 92.7% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.a.log.gz 2023-07-14T17:47:22.400 INFO:teuthology.orchestra.run.smithi088.stderr: 92.8% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.g.log.gz 2023-07-14T17:47:22.400 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mon.a.log 2023-07-14T17:47:22.400 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.j.log: /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mon.a.log: 94.6% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.j.log.gz 2023-07-14T17:47:22.402 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2023-07-14T17:47:22.403 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.i.log: /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T17:47:22.404 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.7.log: /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mon.b.log: 92.7% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.k.log.gz 2023-07-14T17:47:22.404 INFO:teuthology.orchestra.run.smithi107.stderr: 91.4% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.e.log.gz 2023-07-14T17:47:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.admin.log: 91.5% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T17:47:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2023-07-14T17:47:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/ceph-client.0.log: 94.0% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2023-07-14T17:47:22.406 INFO:teuthology.orchestra.run.smithi088.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2023-07-14T17:47:22.407 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.b.log 2023-07-14T17:47:22.407 INFO:teuthology.orchestra.run.smithi107.stderr: 91.1% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.h.log.gz 2023-07-14T17:47:22.407 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mgr.y.log 2023-07-14T17:47:22.408 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-volume.log: 91.3% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.c.log.gz 2023-07-14T17:47:22.408 INFO:teuthology.orchestra.run.smithi178.stderr: 92.6% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.f.log.gz 2023-07-14T17:47:22.408 INFO:teuthology.orchestra.run.smithi178.stderr:gzip -5 --verbose -- /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.l.log 2023-07-14T17:47:22.411 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.b.log: /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mgr.y.log: 91.5% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mgr.y.log.gz 2023-07-14T17:47:22.416 INFO:teuthology.orchestra.run.smithi178.stderr:/var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.l.log: 91.5% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T17:47:22.417 INFO:teuthology.orchestra.run.smithi178.stderr: 91.0% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.i.log.gz 2023-07-14T17:47:22.424 INFO:teuthology.orchestra.run.smithi178.stderr: 92.6% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.l.log.gz 2023-07-14T17:47:22.500 INFO:teuthology.orchestra.run.smithi107.stderr: 92.7% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mds.b.log.gz 2023-07-14T17:47:22.538 INFO:teuthology.orchestra.run.smithi178.stderr: 92.0% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mon.c.log.gz 2023-07-14T17:47:22.539 INFO:teuthology.orchestra.run.smithi107.stderr: 92.1% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mon.b.log.gz 2023-07-14T17:47:22.539 INFO:teuthology.orchestra.run.smithi088.stderr: 89.1% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mgr.x.log.gz 2023-07-14T17:47:22.635 INFO:teuthology.orchestra.run.smithi178.stderr: 92.4% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.8.log.gz 2023-07-14T17:47:22.649 INFO:teuthology.orchestra.run.smithi178.stderr: 92.8% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.11.log.gz 2023-07-14T17:47:22.652 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.3.log.gz 2023-07-14T17:47:22.659 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.0.log.gz 2023-07-14T17:47:22.661 INFO:teuthology.orchestra.run.smithi107.stderr: 92.5% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.10.log.gz 2023-07-14T17:47:22.672 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.9.log.gz 2023-07-14T17:47:22.707 INFO:teuthology.orchestra.run.smithi107.stderr: 92.3% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.7.log.gz 2023-07-14T17:47:22.708 INFO:teuthology.orchestra.run.smithi107.stderr: 92.4% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.4.log.gz 2023-07-14T17:47:22.712 INFO:teuthology.orchestra.run.smithi178.stderr: 92.2% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.2.log.gz 2023-07-14T17:47:22.726 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.6.log.gz 2023-07-14T17:47:22.749 INFO:teuthology.orchestra.run.smithi178.stderr: 92.4% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.5.log.gz 2023-07-14T17:47:22.751 INFO:teuthology.orchestra.run.smithi178.stderr: 2023-07-14T17:47:22.751 INFO:teuthology.orchestra.run.smithi178.stderr:real 0m0.378s 2023-07-14T17:47:22.751 INFO:teuthology.orchestra.run.smithi178.stderr:user 0m1.402s 2023-07-14T17:47:22.751 INFO:teuthology.orchestra.run.smithi178.stderr:sys 0m0.065s 2023-07-14T17:47:22.774 INFO:teuthology.orchestra.run.smithi107.stderr: 92.1% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-osd.1.log.gz 2023-07-14T17:47:22.776 INFO:teuthology.orchestra.run.smithi107.stderr: 2023-07-14T17:47:22.776 INFO:teuthology.orchestra.run.smithi107.stderr:real 0m0.403s 2023-07-14T17:47:22.776 INFO:teuthology.orchestra.run.smithi107.stderr:user 0m1.551s 2023-07-14T17:47:22.777 INFO:teuthology.orchestra.run.smithi107.stderr:sys 0m0.071s 2023-07-14T17:47:22.891 INFO:teuthology.orchestra.run.smithi088.stderr: 90.9% -- replaced with /var/log/ceph/b462a556-226c-11ee-9b33-001a4aab830c/ceph-mon.a.log.gz 2023-07-14T17:47:22.893 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-07-14T17:47:22.893 INFO:teuthology.orchestra.run.smithi088.stderr:real 0m0.524s 2023-07-14T17:47:22.893 INFO:teuthology.orchestra.run.smithi088.stderr:user 0m1.822s 2023-07-14T17:47:22.893 INFO:teuthology.orchestra.run.smithi088.stderr:sys 0m0.074s 2023-07-14T17:47:22.894 INFO:tasks.cephadm:Archiving logs... 2023-07-14T17:47:22.895 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337974/remote/smithi088/log 2023-07-14T17:47:22.902 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T17:47:23.252 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337974/remote/smithi107/log 2023-07-14T17:47:23.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T17:47:23.502 DEBUG:teuthology.misc:Transferring archived files from smithi178:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337974/remote/smithi178/log 2023-07-14T17:47:23.503 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T17:47:23.745 INFO:tasks.cephadm:Removing cluster... 2023-07-14T17:47:23.746 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b462a556-226c-11ee-9b33-001a4aab830c --force 2023-07-14T17:47:24.035 INFO:teuthology.orchestra.run.smithi088.stdout:Deleting cluster with fsid: b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:47:24.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b462a556-226c-11ee-9b33-001a4aab830c --force 2023-07-14T17:47:24.788 INFO:teuthology.orchestra.run.smithi107.stdout:Deleting cluster with fsid: b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:47:25.238 DEBUG:teuthology.orchestra.run.smithi178:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b462a556-226c-11ee-9b33-001a4aab830c --force 2023-07-14T17:47:25.538 INFO:teuthology.orchestra.run.smithi178.stdout:Deleting cluster with fsid: b462a556-226c-11ee-9b33-001a4aab830c 2023-07-14T17:47:26.000 INFO:tasks.cephadm:Removing cephadm ... 2023-07-14T17:47:26.001 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T17:47:26.022 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T17:47:26.044 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T17:47:26.065 INFO:tasks.cephadm:Teardown complete 2023-07-14T17:47:26.066 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-14T17:47:26.076 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-14T17:47:26.076 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T17:47:26.079 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T17:47:26.088 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T17:47:29.271 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-14T17:47:29.283 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-14T17:47:29.283 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T17:47:29.288 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T17:47:29.290 DEBUG:teuthology.orchestra.run.smithi178:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T17:47:29.304 INFO:teuthology.orchestra.run.smithi088.stderr:bash: ntpq: command not found 2023-07-14T17:47:29.307 INFO:teuthology.orchestra.run.smithi107.stderr:bash: ntpq: command not found 2023-07-14T17:47:29.309 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T17:47:29.309 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2023-07-14T17:47:29.310 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 8 +960us[ +889us] +/- 103ms 2023-07-14T17:47:29.310 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 10 -799us[ -870us] +/- 61ms 2023-07-14T17:47:29.310 INFO:teuthology.orchestra.run.smithi088.stdout:^* hv03.front.sepia.ceph.com 3 6 377 8 +219us[ +148us] +/- 70ms 2023-07-14T17:47:29.310 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:47:29.311 INFO:teuthology.orchestra.run.smithi178.stderr:bash: ntpq: command not found 2023-07-14T17:47:29.313 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T17:47:29.313 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2023-07-14T17:47:29.313 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 +1136us[+1024us] +/- 103ms 2023-07-14T17:47:29.313 INFO:teuthology.orchestra.run.smithi107.stdout:^* hv02.front.sepia.ceph.com 3 6 377 7 -653us[ -766us] +/- 61ms 2023-07-14T17:47:29.313 INFO:teuthology.orchestra.run.smithi107.stdout:^- hv03.front.sepia.ceph.com 3 6 377 7 +392us[ +392us] +/- 70ms 2023-07-14T17:47:29.314 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:47:29.315 INFO:teuthology.orchestra.run.smithi178.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T17:47:29.315 INFO:teuthology.orchestra.run.smithi178.stdout:=============================================================================== 2023-07-14T17:47:29.316 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 10 +1000us[ +905us] +/- 103ms 2023-07-14T17:47:29.316 INFO:teuthology.orchestra.run.smithi178.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 8 -815us[ -815us] +/- 61ms 2023-07-14T17:47:29.316 INFO:teuthology.orchestra.run.smithi178.stdout:^* hv03.front.sepia.ceph.com 3 6 377 9 +250us[ +155us] +/- 70ms 2023-07-14T17:47:29.316 INFO:teuthology.orchestra.run.smithi178.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T17:47:29.317 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-14T17:47:29.327 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-14T17:47:29.328 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-14T17:47:29.338 DEBUG:teuthology.orchestra.run.smithi088:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T17:47:29.454 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T17:47:29.555 DEBUG:teuthology.orchestra.run.smithi178:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T17:47:29.655 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T17:47:29.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:47:29.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T17:47:29.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:47:29.724 DEBUG:teuthology.orchestra.run.smithi178:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T17:47:29.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:47:29.760 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-14T17:47:29.770 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-14T17:47:29.782 INFO:teuthology.task.internal:Duration was 1730.753980 seconds 2023-07-14T17:47:29.782 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-14T17:47:29.797 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-14T17:47:29.797 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T17:47:29.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T17:47:29.803 DEBUG:teuthology.orchestra.run.smithi178:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T17:47:29.845 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T17:47:29.848 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T17:47:29.853 INFO:teuthology.orchestra.run.smithi178.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T17:47:30.233 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-14T17:47:30.234 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2023-07-14T17:47:30.234 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T17:47:30.256 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2023-07-14T17:47:30.256 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T17:47:30.278 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi178.front.sepia.ceph.com 2023-07-14T17:47:30.279 DEBUG:teuthology.orchestra.run.smithi178:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T17:47:30.302 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-14T17:47:30.303 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T17:47:30.306 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T17:47:30.321 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T17:47:30.391 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-14T17:47:30.391 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T17:47:30.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T17:47:30.396 DEBUG:teuthology.orchestra.run.smithi178:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T17:47:30.508 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-14T17:47:30.519 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-14T17:47:30.519 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T17:47:30.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T17:47:30.577 DEBUG:teuthology.orchestra.run.smithi178:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T17:47:30.605 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-14T17:47:30.624 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T17:47:30.627 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T17:47:30.629 DEBUG:teuthology.orchestra.run.smithi178:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T17:47:30.651 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2023-07-14T17:47:30.653 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2023-07-14T17:47:30.671 INFO:teuthology.orchestra.run.smithi178.stdout:kernel.core_pattern = core 2023-07-14T17:47:30.719 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T17:47:30.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:47:30.736 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T17:47:30.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:47:30.773 DEBUG:teuthology.orchestra.run.smithi178:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T17:47:30.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:47:30.790 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-14T17:47:30.808 INFO:teuthology.task.internal:Transferring archived files... 2023-07-14T17:47:30.809 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337974/remote/smithi088 2023-07-14T17:47:30.810 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T17:47:30.870 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337974/remote/smithi107 2023-07-14T17:47:30.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T17:47:30.913 DEBUG:teuthology.misc:Transferring archived files from smithi178:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337974/remote/smithi178 2023-07-14T17:47:30.914 DEBUG:teuthology.orchestra.run.smithi178:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T17:47:30.958 INFO:teuthology.task.internal:Removing archive directory... 2023-07-14T17:47:30.959 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T17:47:30.961 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T17:47:30.965 DEBUG:teuthology.orchestra.run.smithi178:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T17:47:31.004 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-14T17:47:31.022 INFO:teuthology.task.internal:Not uploading archives. 2023-07-14T17:47:31.023 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-14T17:47:31.034 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-14T17:47:31.035 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T17:47:31.038 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T17:47:31.040 DEBUG:teuthology.orchestra.run.smithi178:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T17:47:31.052 INFO:teuthology.orchestra.run.smithi088.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 17:47 /home/ubuntu/cephtest 2023-07-14T17:47:31.054 INFO:teuthology.orchestra.run.smithi107.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 17:47 /home/ubuntu/cephtest 2023-07-14T17:47:31.060 INFO:teuthology.orchestra.run.smithi178.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 17:47 /home/ubuntu/cephtest 2023-07-14T17:47:31.061 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-14T17:47:31.072 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-14T17:47:31.098 INFO:teuthology.nuke:Checking targets against current locks 2023-07-14T17:47:36.135 DEBUG:teuthology.nuke:shortname: smithi088 2023-07-14T17:47:36.136 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T17:47:41.174 DEBUG:teuthology.nuke:shortname: smithi107 2023-07-14T17:47:41.175 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T17:47:41.181 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337974', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:08:25.060848', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-14T17:47:41.204 DEBUG:teuthology.nuke:shortname: smithi178 2023-07-14T17:47:41.205 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T17:47:41.211 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337974', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:08:25.061798', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-14T17:47:41.249 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi178.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7337974', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:08:25.059060', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-14T17:47:41.259 INFO:teuthology.orchestra.console:Power off smithi088 2023-07-14T17:47:41.262 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T17:47:41.287 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:47:41.321 INFO:teuthology.orchestra.console:Power off smithi107 2023-07-14T17:47:41.322 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T17:47:41.348 INFO:teuthology.orchestra.console:Power off smithi178 2023-07-14T17:47:41.349 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T17:47:41.362 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:47:41.378 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:47:45.363 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:47:45.392 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:47:45.404 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:47:50.479 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:47:50.506 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:47:50.518 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:47:55.596 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:47:55.621 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:47:55.633 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:48:00.711 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:48:00.736 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:48:00.748 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:48:05.825 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:48:05.849 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:48:05.861 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:48:06.063 INFO:teuthology.orchestra.console:Power off for smithi107 completed 2023-07-14T17:48:10.939 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:48:10.977 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:48:16.052 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:48:16.091 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi178.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:48:16.303 INFO:teuthology.orchestra.console:Power off for smithi178 completed 2023-07-14T17:48:21.172 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:48:26.288 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:48:31.404 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:48:36.520 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:48:41.633 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T17:48:41.845 INFO:teuthology.orchestra.console:Power off for smithi088 completed 2023-07-14T17:48:41.953 INFO:teuthology.run:Summary data: description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}} duration: 1730.753980398178 failure_reason: 'Command failed on smithi088 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''ceph fs subvolumegroup pin cephfs _nogroup random 0.01''"' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ff352723e68348babba8dbca4ff1b611 status: fail success: false 2023-07-14T17:48:41.954 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-14T17:48:47.053 INFO:teuthology.run:FAIL